Summary: GDPR updates part 2
Release Date: Tue, May 29, 2018 at 2:38 PM UTC
Release Author: Henrik Levkowetz

This is the second in a small series of releases to address compliance
with the European GDPR (General Data Protection Regulation).

This release provides two data migrations to polulate email address origin
and other trace information, to make it possible to differentiate between
that personal information which is to be treated as Consent-based, that
which falls under Legitimate Interest.

Coverage

Summary: GDPR updates part 1
Release Date: Tue, May 29, 2018 at 11:10 AM UTC
Release Author: Henrik Levkowetz

This is the first in a small series of releases to address compliance with
with the European GDPR (General Data Protection Regulation).

This release provides necessary GUI and internal changes, but lacks some
admin utilities, and also data migration routines which takes some time to
run, and therefore will be packaged in a separate release and applied
separately.

From the commit log:

• Updated the personal information page with reviewed text from legal
  counsel. Fixes issue #2503.

• Removed the Person.address field, which is not being used. This was a
  legacy from the 2001 perl-based datatracker tables. Fixes issue #2504.

• Added django-simple-history and replaced the old (and unused)
  PersonHistory class with a history=HistoricalRecords() field on Person.
  Added the needed migrations and changes to admin, resources, and settings.
  Related to issues #2505 and #2507.

• Added a new field name_from_draft to Person, to hold the name field
  equivalent as captured from drafts, in case name has been modified by the
  user and we're asked to remove that info under GDPR. Added history for
  Email, and also an origin field to capture from where we got an email
  address (draft name, username, meeting registration, etc.) Added a
  log.assertion() to Email.save() in order to ensure we don't create any
  email without setting origin.

• Added origin information to all places where we create email address
  entries.

• Removed all references to the removed Person.affiliation field.

• Added email origin information to some function calls that needed it.

• Overwrite earlier email origin when we've picked up the address from a
  submission.

• Added a consent field to the Person model.

• Disallow profile changes without consent given. Together with previous
  commits this fixes issues #2505 and #2507.

• Added another category of personal information to the
  personal-information page, after review of personal information in the
  code. Completes issue #2501.

• Added a dagger at the end of some fields in the account data forms to
  signify consent-based fields, and made the consent field required.

• Simplified the email.origin assignment code for outgoing liaisons.

• Fixed a long-standing bug in the liaison.name() code.

• Added assingment of the person.name_from_draft field on draft
  submission.

• Updated the edit_profile template with information about consent-based
  fields. Fixes issue #2502.

Coverage

Release Date: Thu, May 24, 2018 at 3:53 PM UTC
Release Author: Henrik Levkowetz

This is a bugfix release which also clears the slate for the upcoming
GDPR-related release. From the commit log:

• Added a meeting helper function update_important_dates(), and a management
  command update_important_dates to update the important dates for a meeting
  if the offsets in the MeetingDateName table has been changed.

• Added some text to the important dates page for opensched, on request
  from the IESG via Stephanie.

• Merged in [15153] from rjsparks@nostrum.com:
  Changed how the mail utilities handle lists in their override and extra
  arguments. Added a test for send_mail_preformatted exercising override and
  extra.

• Merged in [15152] from rjsparks@nostrum.com:
  Send email when the responsible AD for a conflict review is changed.
  Include the stream manager and stream steering group when the responsible
  AD changes or when the telechat date changes. Fixes #2497.

• Merged in [15151] from rjsparks@nostrum.com:
  Provide an email group button for WG and RG documents. Fixes #2512.

• Merged in [15145] from rjsparks@nostrum.com:
  Secretariat requested tweak to the request for minutes template.

• Updated the non-WG mailing list page to include advertised lists for
  closed WGs, on request from the secretariat.

Coverage

Release Date: Tue, May 8, 2018 at 6:37 AM UTC
Release Author: Henrik Levkowetz

This release fixes a number of bugs, and adds a view that makes it easier
for the secretariat to generate the meeting minutes request emails.

• Merged in [15125] from rjsparks@nostrum.com:
  Added a view that generates the send minutes email for the secretariat.
  Fixes #2493.

• Merged in [15124] from rjsparks@nostrum.com:
  Provide a unicode member for ImportantDate.

• Merged in [15123] from rjsparks@nostrum.com:
  Use importantdates for material submisssion cutoff and revision if they
  exist. Fixes #2514.

• Merged in [15119] and [15120] from rjsparks@nostrum.com:
  Change stream state to replaced when an ISE, IAB, or IRTF document is
  replaced. Fixes #1963.

• Merged in [15114] from rjsparks@nostrum.com:
  Restored a test that was unintentionally removed. Removed unused
  templates. Related to #2513.

• Merged in [15113] from rjsparks@nostrum.com:
  Removed the already disabled Make RFC functionality from /secr/drafts.
  Fixes #2513.

  Merged in [15112] from rjsparks@nostrum.com:
  Removed the announce functionality from /secr/drafts. Addresses part of
  #2513.

• Merged in [15111] from rjsparks@nostrum.com:
  Removed the 'replaces' functionality from /secr/drafts. Addresses
  part of #2513.

• Merged in [15110] from rjsparks@nostrum.com:
  Made settings.URL_REGEXPS.name non-greedy. Added a test to exercise the
  bug at #2429. Fixes #2429.

• Fixed an incorrect file name in a log message.

Coverage

Release Date: Wed, May 2, 2018 at 10:42 AM UTC
Release Author: Henrik Levkowetz

This bugfix release fixes problems with IESG ballot clearing, marks dead
code as unreachable for later removal, adds new tests to increase test
coverage, and fixes some issues with the test suite. It also tweaks the
roles which are permitted to request document publication, loosens the
need for trailing slashes for some URLs, and adds a Recent drafts page.

• Merged in [15084] from rjsparks@nostrum.com:
  Corrected the implementation of clear_ballot. Added a test for it.
  Restored functionality to ballot popups that was missing because a
  template variable was not passed in.

• Added a recent drafts page, showing drafs submitted the last 7 days.

• Merged in [15083] from rjsparks@nostrum.com:
  Removed memoize from latest_event.

• Merged in [15080] from rjsparks@nostrum.com:
  Marked a couple of functions as unreachable.

• Reverted a change to UserFactory to avoid nepalese names. The issue is
  bigger, and needs a different resulution. Changed the url pattern that
  used \w to match names to instead use a more inclusive regex.

• Fixed a PyQuery property access that should have been a method call.

• Merged in [15076] from rjsparks@nostrum.com:
  Added a test for meeting/requests.

• Tweaked the IPR factory to be less likely to produce invalid patent
  info.

• Merged in [15075] from rjsparks@nostrum.com:
  Improved testing for docs_for_ad.

• Merged in [15074] from rjsparks@nostrum.com:
  Simplified docs_in_iesg_process. Added a test for it.

• Merged in [15073] from rjsparks@nostrum.com:
  Rearranged ietf/doc/urls.py to allow matching some special pages under
  /doc without a trailing slash. Adjusted several of those special URLs to
  work without the trailing slash.

• Added permission for a document shepherd to request publication. Fixes
  issue #2509.

Coverage

Summary: IPR patent info field fixes, IESG position API enhancement, and more.
Release Date: Mon, April 23, 2018 at 12:28 PM UTC
Release Author: Henrik Levkowetz

This release enhances the IESG position API to send notification emails on
changes, widens some IPR disclosure form fields to accept more text, and
fixes a patent application number validation issue for IPR forms. It adds
some useful links and a new filtering option to the past meetings page,
updates document stream status indications so as to be less prone to
misinterpretation, addresses an issue with session request forms, and fixes
an issue with I-D expirations and another related to ordering of documents
on IESG agendas.

From the commit log:

• Set file upload permissions to 0644 to avoid problems with photo access.

• Increased the length of the patent title in IPR forms

• Fixed a typo. Fixes issue #2500 which caused different document ordering
  on the IESG agenda compared with the IESG agenda documents page.

• Increased the size of the patent_notes form field, and tweaked the
  validate_patent_number validator to accept US application numbers
  containing a series number and forward slash.

• Merged in [15055] from rcross@amsl.com:
  Fixed an issue where user can make two session request submissions,
  corrupting data. Also, ignore deleted session requests in edit view.

• Merged in [15047] from rcross@amsl.com:
  Reduced the get_meeting buffer to handle earlier shceduling requests date.

• Added origin indication to a group of stats templates to make it easier
  to find the origin of stats configurations.

• Added group links to the past meetings page, and also important dates
  links to IETF meetings with important-date pages.

• Added IETF to past meetings filter pane, in order to be able to filter
  out interims.

• Fixed a problem with the expire-ids code, and added logging to catch
  future problems.

• Modified the IESG position setting API to send email messages about
  positions set.

• Tweaked the wording of coverage_changes output.

• Fixed an issue where simplified sorting of DocEvents resulted in some
  ballot icons not being shown on the IESG agenda documents page.

• Changed incorrect use of 404 status code in api to 400. Fixes issue
  #2498.

• Removed what could be a misleading RFC from the stream indication in
  document search results.

Coverage

Summary: Document search on IRTF state, performance improvements
Release Date: Tue, April 10, 2018 at 4:33 PM UTC
Release Author: Henrik Levkowetz

This release brings document search support for filtering on IRTF states,
courtesy of Mat Ford, and further performance improvements to the IESG
agenda document page and document search pages. There are also improvements
to the testing infrastructure, HTML cleanup, some added API documentation,
and a number of bugfixes, as follows:

• Merged in [15012] from ford@isoc.org:
  Added IRTF State option to Additional Search Criteria pulldown.

• Included django.utils.html.escape() in the linkify() code, and removed
  it from various templates, to make sure escape is consistently applied
  before linkify. Fixes issue #2492.

• Tweaked the error return from Document.get_document_content() to be
  more helpful during development.

• Fixed various html issues.

• Added more information about personal API keys to the API help page.

• Tweaked the test runner to more consistently force the test coverage
  tests to be run last.

• Changed calls to .related_that*() and .relations_that*() to use tuples
  rather than lists (a slight optimization).

• Silenced another deprecation warning from a library.

• Reduced the max number of results returned by document searches from 500
  to 200, for a better user experience. A faster return and a search term
  tweak to pinpoint the desired search result is better than a long wait and
  then maybe having to tweak the search terms anyway.

• Use .assertIn(...) in tests instead of .assertTrue(x in y) as the
  failure messages are much more informative

• Changed the signature of some document methods to not accept lists (but
  still accept tuples) in order to make caching possible. Added @memoize to
  some methods.

• Added another prefetch to search doc query results.

• Imposed sort order (most recent docs first) on document search results,
  in order to retain the most recent docs in case we have to chop the result
  list due too many hits.

• Tweaked the django_cprofile_middleware patch to use a query arg of
  psort instead of sort, to avoid conflict with app query args sort.

• Added back the on-agenda calendar links to document lists, now in an
  implementation with better performance characteristics.

• Updated minimum requirements for bootstrap3, to work with django 1.11

• Fixed an issue where the change of slide naming between IETF 96 and 97
  caused the datatracker to not provide correct link to pre-IETF-97
  materials. Thanks to tjw.ietf@gmail.com for the heads-up.

• Merged in [14970] from rjsparks@nostrum.com:
  Factories for the ipr app. Coverage of ietf.ipr.views.ajax_search, and
  a factory-driven test of ietf.ipr.views.edit.

• Enhanced debug.py: made the mark() function indicate file and line
  number it was called from. Added if debug: conditions around some other
  function actions.

• Another round of iesg page optimizations. Reduced additional instances
  of individual lookups to a single lookup followed by annotation of document
  objects. Reduced several instances of repeating the same work multiple
  times (queries for telechat documents, queries for telechat dates).

• Fixed a number of tests to deal correctly with model method caching.

• Changed the ballot_icon template tag to use an already found ballot if
  available, rather than doing lookups to find it (twice!).

• Added a patch for debug mode, to add a filtering option to the
  django-cprofile-middleware.

• Be more specific with bootstrap's layout classes for small devices in
  the base template, to avoid the bottom links and footer overlaying the
  bottom of the content area. Fixes a problem with entering login
  information on small devices.

Coverage

Summary: Performance improvements
Release Date: Mon, April 2, 2018 at 11:55 AM UTC
Release Author: Henrik Levkowetz

This release brings performance improvements, focused in particular on the
IESG agenda documents page. Some of the changes will also provide
performance benefits for other pages, in particular pages that lists
documents, such as search result pages and WG/RG document lists. All in
all, the changes speed up the rendering of /iesg/agenda/documents with a
factor of about 2. The page generation time is still roughly proportional
to the number of documents on the page, so a large number of documents on
future IESG telechats will still give a slow page. More substantial
refactoring will probably be needed for further speedups.

From the commit log:

• Added an index for (doc, type) on DocEvent to speed up latest_event()
  queries.

• Changed the reviewed_by_teams list to provide acronyms directly, to avoid
  group lookups during template rendering.

• Changed telechat_page_count() to accept a list of documents if such a list
  has already been generated, to avoid double work.

• Provided telechat page counts to the template from the view, instead of
  repeated (costly) filtering through the telechat_page_count template
  filter in the template, which caused new, redundant document lookups every
  time. Removed the telechat_page_count template filter altogether, due to
  its cost.

• Tweaked some lookups in fill_in_document_table_attributes().

• Added terms to the select_related() list for documents in IESG
  agenda_documents().

• Added a get_active_ads() utility function that uses the cache to reduce
  database access, and replaced database queries in various places with
  calls to get_active_ads().

• Moved some utility functions so as to reduce the likelihood of import

• loops.

• Changed some instances of .slug to _id, to avoid unnecessary database
  lookups.

• Added a profiling middleware to development settings, if installed.

Coverage

Release Date: Fri, March 30, 2018 at 7:11 PM UTC
Release Author: Henrik Levkowetz

Automation API endpoint: /api/meeting/session/video/url

This release provides provides a new API endpoint to let video urls be
associated with sessions, intended for use by Meetecho to update the
datatracker with sesssion recording links as soon as a recording has been
uploaded to the YouTube (or equivalent). It also introduces a new
Recording Manager role, used to authorize the API access. A short
description of the API is available at
https://datatracker.ietf.org/api#session-video-url-api

Coverage

Release Date: Wed, March 28, 2018 at 8:54 PM UTC
Release Author: Henrik Levkowetz

This release fixes a number of blemishes and cleans the slate for merging
in a meetecho API branch.

• Fixed an issue which could result in server 500 errors for paths like
  /meeting/99/session/plenary.

• Removed superfluous stats pages headers which would be repeated with
  different layout on the next line, as part of the graphs.

• Added a test for IPR edit.

• Filled in the IPR edit form, section V, with patent information derived
  from the patent_info field, the same way we do for the IPR update form.

• Added a pip version restriction to requirements.txt, as there seems to be
  issues with pip 9.0.2 and 9.0.3; they seem to leave dross when down- and
  upgrading packages.

• Don't show links to individual meeting stats pages when there's no
  content (meeting 71 and earlier currently has no attendance records).

• Added meeting dates to the meeting stats overview graph.

• Changed the name of a loop variable so as not to obscure a previous
  local variable.

• Don't show future meetings in the meeting stats overview.

• Changed settings so that cache keys used by memcached includes the
  release version, in order to avoid stale and incorrect cache content on new
  release deployment. Made it easier to pick out cache key composition by
  normalizing the variable name. Adjusted cache prefix and the composition
  of some keys to have a unique and consistent cache key prefix.

• Use the actual defaults from settings.py when showing the defaults on
  /accounts/settings/.

Coverage