PBS video impression tracking

As a followup to the event tracking issue #800, we're ready to implement the injection of additional tracking into VAST XML. Initially this is just to count impressions, but will be expanded to additional video events at some point. There are two scenarios, both of which are required and could be happening at the same time:

Server-side VAST - this scenario covers the video XML is returned in the response from a server-side adapter. If that bidder allows PBS to modify their VAST, the server then injects an tag into the VAST, pointing to a prebid server event string, and then caches the modified results.

Here's a flow diagram showing how tracking will work. Changed items are in red.

Client-side VAST - this scenario occurs when the video XML is returned in the response from a client-side adapter. PBJS forwards the XML to PBS on a new endpoint that instructs PBS to update the XML and cache it. If that bidder allows PBS to modify their VAST, the server then injects an tag into the VAST, pointing to a prebid server event string, caches the modified results and responds to the client.

Proposed modifications

Add a new bidder-level flag in PBS that allows a given bidder to turn off our ability to modify their VAST. e.g. if bidderA doesn't want us to modify their VAST, the host company won't have complete analytics.

We propose a new bidder configuration property modifyingVastXmlAllowed, with the default being false.

When PBS receives VAST from each bidder, it checks the bidder flag and whether the account supports events. If so, it inserts an <impression> tag before storing to PBC

The contents of the <impression> tag are pulled from a new event.url-template property that has macros that need to be resolved. e.g.

event:
    url-template: "/event?t=imp&b=%s&f=b&a=%s"

where b=BIDID, a=ACCOUNT

Algorithm for updating the VAST. (originally this was over-simplified. Changed May 13, 2021)

Scan the 'adm' on video bids:

If adm contains <InLine> (case insensitive), it's "Inline.
Otherwise, if adm contains <Wrapper> (case-insensitive), it's a "wrapper".
Otherwise if adm is not empty but doesn't contain either Inline or Wrapper, don't modify the results.
Otherwise if adm is empty and nurl is present, create a VAST wrapper:

<VAST version=\"3.0\"><Ad><Wrapper>
<AdSystem>prebid.org wrapper</AdSystem>
<VASTAdTagURI><![CDATA[" + bidNurl + "]]></VASTAdTagURI>
<Creatives></Creatives>
</Wrapper></Ad></VAST>

Otherwise if neither adm nor nurl are present, it's an invalid video bid.

Once we know what type of VAST is present:

Inline VAST - search for <Impression>. Add PBS Impression tag(s) after existing tag. If no <impression> tag is found, it's an unexpected error and this VAST is suspicious. Don't add any PBS Impression tags.
Wrappers
- Look for an impression pixel on the Wrapper
- If impression pixel present, insert <Impression> tags at the same level
- If impression not present, refer the xsd (for VAST version in the declaration) and find the parent element for impression pixels. Add the impression pixels one level below the parent element
- If parent element not present, reject the VAST and the corresponding bid.
VAST URL - create a PBS-generated wrapper that includes one or more Impression tags.

Client-Side VAST Caching, including PBJS changes

Unlike server-side video, the VAST XML coming into the browser didn't go through Prebid Server, so will not have the tracking strings added. Prebid.js has configuration that allows the publisher to initiate "client-side caching". In order to modify the VAST XML, the page will refer to a new end point on Prebid Server that will perform the same modifications as if it had come through the server.

pbjs.setConfig({
    cache: {
        url: "https://prebid-server.rubiconproject.com/vtrack?a=ACCOUNT",
        vasttrack: true
    }
});

Where /vtrack is a new endpoint that causes the request to go to PBS where the steps 1-3 above will be applied.

If the vasttrack parameter is true, Prebid.js will add a couple of parameters to the POST XML to the specified endpoint with this JSON

{"puts":[{
    "bidid": BIDID,             // new parameter
    "bidder": "BIDDER",    // new parameter
    "type":"xml",
    "value":"<VAST…/VAST>",
    "ttlseconds":3600
}]}

PBS doesn't currently have a /vtrack endpoint -- we shouldn't use /cache because PBC implements that one. The new endpoint would use a similar code path as for server-side VAST:

parse the posted VAST and pull out the bidid and bidders parameters
use the bidid and the account id to create the additional field in the VAST
POST the modified VAST to Prebid Cache, wait for the results from PBC, and forward them to the client. e.g. {"responses":[{"uuid":"94531ab8-c662-4fc7-904e-6b5d3be43b1a"}]}

PBS should validate the arguments supplied with /vtrack: if ACCOUNT, BIDID, or BIDDER aren't supplied, it should reject the /vtrack request with 400. This should get caught when the page is being tested.

If the ACCOUNT doesn't exist and PBS is enforcing accounts, that should also result in a 400.

PBS will process /vtrack similar to the server-side VAST response, inserting impression tracking when:

account ID allows events
the bidder allows VAST modification

Closed Item:

We could combine the "/vtrack" into "/cache" but that could make load balancer configuration more complicated. We discussed in the PBS meeting last week and agreed a separate endpoint is the way to go.

For load balancer reasons, I'm starting to prefer a separate endpoint /vtrack.

@bretg

Hi @bretg !
From PBS configuration point of view i propose to use instead of

events:
    urltemplate: "https://prebid-server.rubiconproject.com/event?t=imp&b=%s&f=b&a=%s"

event:
    url-template: "/event?t=imp&b=%s&f=b&a=%s"

So, the changes:

prefix for the property - event.* since PBS-Java has /event endpoint, so can be more acceptable.
name for the property - url-template to be more readable.
value for the property - includes path with query string since both PBS-Go and PBS-Java has ExternalURL configuration property, so it can be simply joined to obtain full url.

@rpanchyk

Thanks @rpanchyk - updated.

Discussed and approved in PBS PMC

@hhhjort

This is done in PBS-Java. Assigning to @hhhjort for PBS-Go implementation.

For the record, here are the PBS-Java PRs that implemented events:

prebid/prebid-server-java#296
prebid/prebid-server-java#298
prebid/prebid-server-java#302 (partially)
prebid/prebid-server-java#403
prebid/prebid-server-java#436
prebid/prebid-server-java#437
prebid/prebid-server-java#450
prebid/prebid-server-java#459 (refactoring)
prebid/prebid-server-java#635
prebid/prebid-server-java#654
prebid/prebid-server-java#804 (doc)

@laurb9

This is largely complete in PBS-Go. @laurb9 noted two remaining things to implement:

wurl insertion in bids
VAST rewriting when pbs caches

@danielguedesb Are you able to continue contributing to add those areas of functionality?

This older ticket did not define the entire functionality needed for events and I can't find another one that does (edit: or maybe #1470 ), so I will add it here from https://docs.google.com/document/d/1ry0X4C2EV-R0pMrm1IQk9BstxaT395UCl3KKqTGa5c8/edit# so we are all on the same page. Let me know if this should be another ticket. I'm starting work on this.

If the account doesn't support events, we're done - no need for any type of event logic.
Otherwise, if the bid request was video and the response is VAST XML:
1. If we're allowed to modify the bidder's VAST, then inject an tag with this URL -- https://PBS_HOST/event?t=imp&b=BIDID&f=b&a=ACCOUNT&ts=TIMESTAMP&bidder=BIDDER&int=INTEGRATION
2. If ext.prebid.cache.vastXml is specified, then cache the VAST in PBS
else // not video
1. If bid caching is turned on (ext.prebid.cache.bids)
  1. If the event configuration is on for this account/channel combination or if the ext.prebid.events object is defined in the original request, then add wurl to bids cached in PBC.
    URL is https://PBS_HOST/event?t=win&b=BIDID&f=i&a=ACCOUNT&ts=TIMESTAMP&bidder=BIDDER&int=INTEGRATION
  2. else if it's a PG bid, then add wurl to bids cached in PBC but with x=0
    https://PBS_HOST/event?t=win&b=BIDID&f=i&a=ACCOUNT&ts=TIMESTAMP&bidder=BIDDER&int=INTEGRATION&x=0
2. Finally, consider setting seatbid[].bid[].ext.prebid.events.win and seatbid[].bid[].ext.prebid.events.imp
  1. If the event configuration is on for this account/channel combination or if the ext.prebid.events object is defined in the original request, then add response.seatbid[].bid[].ext.prebid.events.{win,imp} to the openrtb output -- https://PBS_HOST/event?t=win&b=BIDID&f=i&a=ACCOUNT&ts=TIMESTAMP&bidder=BIDDER&int=INTEGRATION and https://PBS_HOST/event?t=imp&b=BIDID&f=i&a=ACCOUNT&ts=TIMESTAMP&bidder=BIDDER&int=INTEGRATION
  2. else if it's a PG bid, then add response.seatbid[].bid[].ext.prebid.events.{win,imp} to the openrtb output -- https://PBS_HOST/event?t=win&b=BIDID&f=i&a=ACCOUNT&ts=TIMESTAMP&bidder=BIDDER&int=INTEGRATION&x=0 and https://PBS_HOST/event?t=imp&b=BIDID&f=i&a=ACCOUNT&ts=TIMESTAMP&bidder=BIDDER&int=INTEGRATION&x=0

Let me know if this should be another ticket.

I think we're good with continuing the discussion here.

I'm starting work on this.

Sweet. Thank you.

I've implemented most of the logic in #1597 . The goal was to get the unit tests to a point where we can validate the behavior, then do some refactoring. Some of the tests are currently failing in a non-significant way because of timestamp mismatch (until #1584 is merged) but are covering the behavior we need.

I see a few differences between what PBS-Java does and the algorithm above:

Only the cached VAST is modified, the VAST returned in adm field is not, in returned or cached bids. Is this intentional ? I have copied this behavior in the PR above.
bid caching and vast modification are not exclusive as the else above suggests. PBS-Java will change both VAST and cached bids if enabled. This made sense to me, so that's what I did too.
PBS-Java adds wurl to cached bid if both account and request enabled it. The algorithm says either suffices, I implemented the algo.
int (integration) does not get a value, so I didn't add it yet.

Implemented spec changes to satisfy unit tests

I've updated the PR to what I think is a potentially releasable functionality. I see exchange code is a refactor target, so I'd like to defer the additional PG and channels functionality as an incremental update for later, to avoid more PR conflicts

Add modifying of VAST for video bids and add validation

@laurb9

@laurb9 - created a somewhat modified version of your truth table at the bottom of https://docs.google.com/document/d/1ry0X4C2EV-R0pMrm1IQk9BstxaT395UCl3KKqTGa5c8/edit# . The only comment for your version:

The "bid dealID" column should be "bid is PG". PBS-Go doesn't support Programmatic Guaranteed so you don't need to worry about those cases.

Responding to questions in your comment:

request.ext.prebid.events only impacts non-video bids as far as I can tell

right - that was the original set up -- the proposal here is that request.ext.prebid.events overrides the server account event/analytics config for video as well as banner/native.

assume that is a qualified "always"

Removed the word "always".

How do we define a PG bid ? bid.dealID being set ?

What is LINEID ? Is it the GAM lineitem (imp.ID), the bid.dealID or something else ?

from the doc

PG not supported in PBS-Go.

In what scenarios should request (.ext.prebid.events) override account (.events_enabled) ? Because PBS will receive events for an account that has events disabled, and that can be confusing.

The idea is that request.ext.prebid.events is a request-level override to the server-side config.

defer the additional PG and channels functionality as an incremental update for later, to avoid more PR conflicts

PG may not be built in PBS-Go. That would be a fairly major effort and ought to be driven by demand for the feature.

Channels -- PBS-Java has new account level "analytics_config" that defines analytics on a per-channel basis. Unfortunately I don't think the JSON attribute very well named. Currently it's "auction-events", but it's really more a generic flag for analytics support by channel. Will open a separate discussion on whether we want to make changes there. #1470

ext.prebid.events now overrides server config

I updated the code and tests so ext.prebid.events is sufficient to trigger VAST modification as well as banner events response.
I've left out the channels and PG.

Please review the unit tests at
exchange/exchangetest/events-bid-account-off-request-off.json
exchange/exchangetest/events-bid-account-off-request-on.json
exchange/exchangetest/events-bid-account-on-request-off.json
exchange/exchangetest/events-vast-account-off-request-off.json
exchange/exchangetest/events-vast-account-off-request-on.json
exchange/exchangetest/events-vast-account-on-request-off.json

cached wurl tests are in
exchange/eventscachetest

@laurb9

@laurb9 We've merged your events PR. Would you say that PBS-Go now implements this feature or are there still gaps?

Out of the things specced in the document, we are only missing PG and integration channels.

PG was said to be not implemented yet in PBS-Go, so that part can be added when PG functionality is worked on.

Integration channels: PBS-Go has a per-account enable flag for events. PBS-Java has that, plus three additional subflags that can control it for amp, app and web individually for each account if set. This was mentioned here. This, I have not implemented because I was left the impression that channel mapping needed further discussion (#1428, #1470), and the channel wasn't readily available in the auction for me to use.

Add modifying of VAST for video bids and add validation (#1081)

Updated VAST updating algorithm

Add modifying of VAST for video bids and add validation (#1081)

This issue will be closed, as everything requested here has been implemented besides PG. And since PG is a big feature, we decided to branch that off into it's own issue #2312

Navigation Menu

Repository navigation

PBS video impression tracking #1015

Proposed modifications

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PBS video impression tracking #1015

Description

Proposed modifications

Activity

bretg commented on Aug 28, 2019

rpanchyk commented on Aug 29, 2019

bretg commented on Sep 3, 2019

bretg commented on Sep 8, 2019

bretg commented on Nov 8, 2019

bretg commented on Aug 11, 2020

SyntaxNode commented on Oct 16, 2020

laurb9 commented on Nov 12, 2020

SyntaxNode commented on Nov 13, 2020

laurb9 commented on Nov 24, 2020

11 remaining items

laurb9 commented on Dec 16, 2020

bretg commented on Dec 21, 2020

laurb9 commented on Jan 6, 2021

SyntaxNode commented on Jan 21, 2021

laurb9 commented on Jan 23, 2021

bretg commented on May 13, 2021

AlexBVolcy commented on Jul 20, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions