Summary: IETF 101 code sprint, release 2
Release Date: Mon, March 26, 2018 at 1:38 PM UTC
Release Author: Henrik Levkowetz

This release provides improved meeting attendance statistics (under
/stats/meeting/overview/), improved test coverage, additional menu entries,
html template fixes, total page count on the Last Call page, the ability
for ADs to clear ballots themselves without going through the secretariat,
and more. It also changes the default cookie setting for showing or hiding
the duplicate left-hand menu column. (If you don't like the new default, use
the User>Preferences menu item to revert to the old look.)

Again, many thanks to all the contributors!

From the commit log:

• Fixed some issues with agenda and proceedings code for meetings
  numbered 64 and below, actualized by the introduction of meeting records
  for such meetings for stats purposes.

• The template filter keep_spacing can interact badly with wordwrap.
  Removed it from several places, and made text document display more
  consistent. Fixes issue #2481.

• Merged in [14902] from rjsparks@nostrum.com:
  Test coverage improvements for ietf/secr/telechat/views.

• Merged in [14900] from rjsparks@nostrum.com:
  Improvements to test coverage for ietf/secr/drafts/views.py.

• Merged in [14899] from rjsparks@nostrum.com:
  Cleaner charter factory simplifies test code.

• Added submenu to the Statistics menu entry, pointing at the individual
  stats pages, and entries for NomComs and Releases.

• Merged in [14896] from rjsparks@nostrum.com:
  Improved test coverage for ietf/doc/views_charter.py.

• Fixed the base template to have id='content' for the content also when
  not showing the left-hand menu.

• Updated the cookies tests to match the new left_menu default.

• Changed the default setting for the datatracker left-hand menu to
  off. The top menu is always present, and works well.

• Tweaked the document list iCalendar links to not show for presentations
  on days in the past.

• Merged in [14884] from adam@nostrum.com:
  Adding page count to last call page.

• Merged in [14883] from adam@nostrum.com:
  Adding link to documents in last call to the AD dashboard side menu.

• Merged in [14849] from ben@nostrum.com:
  Give ADs the red Clear Ballot button

• Merged in [14880] from rjsparks@nostrum.com:
  Added a Draft test suite.

• Merged in [14879] from kivinen@iki.fi:
  Backfilled IETF meetings starting from 1 and backfilled the attendee
  numbers for them. Modified the overview attendee statistics page to use
  different colors for different venue continents.

• Tweaked the meeting registratoin import to deal more gracefully with
  too long names and email addresses, and to fill in the total attendees
  number for the meeting.

• Tweaked mergeready to accept an additional hold-for-merge file outside
  the repository.

Coverage

Summary: IETF 101 code sprint, release 1
Release Date: Wed, March 21, 2018 at 10:58 AM UTC
Release Author: Henrik Levkowetz

This release contains datatracker bugfixes and enhancements from the IETF-101
Code Sprint. It brings new views to merge duplicate person records, fixes
an annoying bug in downref identification and a bunch of links that didn't
point to the right document, corrects the mime-type for JSON responses, and
brings substantial improvments to test coverage.

Many thanks to everyone who contributed!

• Merged in [14864] from rjsparks@nostrum.com:
  Improved test coverage for ietf.meeting.views.

• Updated Note Well. Fixes issue #2474.

• Tweaked person factory ascii name generation to not include periods
  inside names.

• Tweaked a factory to make duplicate mailing list names less likely in tests

• Test data template tweak.

• Merged in [14863] from rjsparks@nostrum.com:
  Improved test coverage for ietf.utils.draft.

• Merged in [14862] from rcross@amsl.com:
  Add view for merge person records.

• Merged in [14855] from rjsparks@nostrum.com:
  Improved test coverage for ietf.meeting.views.

• Added a test case for reference extraction.

• Merged in [14854] from rjsparks@nostrum.com:
  Improved test coverage over meeting.views.

• Merged in [14851] from housley@vigilsec.com:
  Improve parser for references in Internet-Drafts. Fixes #2360

• Merged in [14850] from rjsparks@nostrum.com:
  Improvements to test coverage in ietf.meeting.views

• Removed premature save of upload document state (doing it after
  verifying that the save to file worked instead). Tweaked materials
  documents revision extraction regexes. Added a test on no change of
  materials rev after failed upload. Corrected the naming of materials in
  make_meeting_test_data. Refined the test crawl of meeting materials pages.

• Added a couple of meeting material crawls to check that internal
  material links aren't broken.

• Check attribute isn't None before accessing sub-attributes.

• Merged in [14845] from rcross@amsl.com:
  Remove unused roles/forms.py.

• Merged in [14844] from adam@nostrum.com:
  Fixing href field for
  https://datatracker.ietf.org/meeting//json.

• Fixed some materials docname-with-rev regexes that were too narrow.

• Merged in [14843] from adam@nostrum.com:
  Updated (deprecated, unregistered) text/json with application/json
  everwhere.

• Merged in [14842] from rcross@amsl.com:
  Add check for deleted session when scheduling. Fixes #2450.

• Fixed a bug in a RegexValidator error message string interpolation.

Coverage

Summary: Session iCalendar links on document lists
Release Date: Sat, March 17, 2018 at 10:42 AM UTC
Release Author: Henrik Levkowetz

This release adds session ical links to group document lists and document
search results, fixes some issues related to upload sanitization and url
resolution, improves performance for some pages, and adds and improves some
tests in the test suite. From the commit log:

• Merged in [14830] from rjsparks@nostrum.com:
  Simple coverage tests for ietf/secr/drafts/reports.py.

• Tweaked the document sanitizer to insert a charset meta tag after
  sanitization.

• Fixed a mistaken change in session urls.

• Tweaked a test case to make it easier to get at failures.

• Made the materials_document() view function more robust in handling
  various materials document names.

• Gave the 404 message a class which will make it easier to pick out the
  message during testing.

• Updated the meeting materials document view to handle urls with
  revisions.

• Updated the meeting document href patterns in settings.py to reflect
  that we now have versioned agendas, minutes, and slides.

• Added session ical links to document lists (WG docs, search results,
  etc.) for documents on upcoming meeting agendas. Reduced the query and
  rendering times of document lists some more through additional
  prefetch_related().

• Added result caching for the Person.email() method.

• Added result caching for a couple of Document methods.

• Added a missing tag to the sanitizer whitelist (telling lxml's Cleaner
  to not clean style with style=False is apparently not always enough).
  Fixes issue #2470.

• Added a debug import.

• Updated a django patch to add origin information to query sets.

Coverage

Summary: Modified HTML upload sanitization
Release Date: Wed, March 14, 2018 at 11:04 AM UTC
Release Author: Henrik Levkowetz

Feedback from wgchairs@ietf.org indicated that stripping out all styling
from uploaded files is too harsh. This release modifies the sanitization to
permit <style> tags in the uploads, and differentiates between fragment
santitization (through the sanitize template filter) and document
santitization. This release also addresses some other issue encountered
with the new upload code, and introduces saving of uploaded files in a
consistent encoding (UTF-8). From the commitlog:

• Added handling for when file magic doesn't return a definitive encoding
  for a file. Added a test case to excercise upload error cases.

• Changed the meeting materials uploads to use the upload file encoding
  found by file-magic when decoding the upload content, and also return
  errors to the user if decoding the upload fails. Fixes issue #2469. This
  will also have the benefit (since we're saving with utf-8 encoding after
  decoding) of having meeting materials consistently stored with a the same
  encoding on the server.

• Added a new argument encoding= to handle_upload_file() in order to be
  able to deal better with various upload encodings.

• Tweaked the mime type validator so it can be called also when there's no
  explicitly required mime types, in order to consistently return mime-type
  and encoding.

• Changed to an empty iterable instead of None in the valid upload
  mime-types settings when there's no required mime type, in order to
  simplify other code.

• Return encoding information to the FileUploadForm when doing mime type
  validation, for later use in decoding.

• Changed html cleaning to differentiate between fragment cleaning and
  document cleaning. Added an lxml-based cleaner for document cleaning, also
  permitting <style> tags (but not external style sheets).

• Changed order of arguments in a test assert for better error legibility.

• Updated PLAN

Coverage

Summary: Sanitization of HTML uploads
Release Date: Mon, March 12, 2018 at 3:00 PM UTC
Release Author: Henrik Levkowetz

During the last few IETF meetings, there have been a few cases of agenda and
minutes uploads that have not worked well, for various reasons. Some have
unintentionally used frames, and failed to include the frame contents; some
have used iframes, which pulls the actual content from elsewhere, which
means it won't actually be saved on the IETF servers and archived. There
has also been issues relating to styling and use of javascript. This shows,
of course, that malicious uploads (even if unintentional) are possible.

Considering this, it seems that a good and general approach would be to do
what is often called sanitization of uploaded html content. (Uploaded
text/plain and markdown documents won't be affected).

This release introduces such sanitization.

The cost of this is that if you upload agendas and minutes in HTML format,
you will need to check the results after upload, to make sure that the
agenda and minutes still captures your intent after the sanitization.

Additionally, there are, as usual, some other features and bugfixes:

• Added sanitization of uploaded html content for session agendas and
  minutes, and did some refactoring of the upload form classes.

• Replaced html sanitization code that called html5lib directly with calls
  to bleach, and upgraded the requirements to let us use the latest html5lib
  and bleach.

• Modified the sanitizer and upload handler to strip not only the tags, but
  also the content of some tags, and to produce valid files (if the content
  is otherwise valid) by wrapping the content in appropriate and

  tags.

• If there are both WG milestone changes and a deletion marked for a
  specific milesone, show both Changed and Deleted labels on the
  milestone review form.

• Changed the handling of the milestone edit form to defer deletion if it is
  requested together with other changes to the same milestone. Instead show
  a warning, and ask that deletion be done without simultaneous changes to
  the milestone.

• Django's urlize filter does not deal well with adjacent parantheses.
  Replaced it with a filter based on bleach.linkify, which does better, and
  also replaced the use of the urlize() function with bleach.linkify(), to
  avoid some exceptions caused by malformed urlize output, exposed by the
  new sanitize_html().

• Removed some dead code.

• Made some per-group and per-session iCalendar links show only if
  occurring in the future.

• Added individual session icalendar links on the group materials pages.

• Added an error message for missing email address in extracted author
  information during automated draft submission.

• Included mailing lists for groups in state bof on the non-WG mailing
  list page, as that seems to match people's expectations better. Thiw was
  triggered by the observation that the iasa20 list was not listed on the
  page.

• Limited the lenght of the ipr document summary to 128, in order to not
  produce overlong message subjects. Fixes a server 500 from 5 Mar 2018.

• The datatracker /html/ URLs don't currently accept revision numbers.
  Fixed the URL in the announcement email for the time being.

• Fixed a typo. Fixes issue #2466.

Coverage

Summary: Various notification email improvements and additional iCalendar links
Release Date: Sun, March 4, 2018 at 4:07 PM UTC
Release Author: Henrik Levkowetz

This release provides a number of bugfixes and enhancements to various
notification email messages; and as a result of resolving ticket #2461
(adding iCalendar links to session scheduling notifications), a refined
ical_ageenda() view function which has permitted the placement of individual
session iCalendar links in a number of useful places. From the commit log:

• Tweaked the routines to find Person records from draft submission
  information to find persons also if the email address case doesn't
  match.

• Added date information to the scheduled session emails, and also a
  link to an ical file for the group sessions. Fixes issue #2461.

• Tweaked some meeting-related admin classes for better search and
  display.

• Fixed a settings typo and updated some settings.

• Renamed some datetime fields from time to modified, to match their
  semantics, and changed them to use auto_now=True. This should fix an issue
  with outdated timestamps on some meeting-related objects.

• Added links for per-session ical entries to the group meetings page and
  to draft pages when a page shows sessions where a draft is on the agenda.

• Refactored ical_agenda() to be more general. Added parameters and code
  to permit generation of per-group and per-session ical pages. Added url
  entries for per-group and per-session ical pages. Changed some parameter
  names to better reflect their semantics ('session'->'acronym').

• Merged in [14715] from rjsparks@nostrum.com:
  Make a more appropriate review message when a BoF has an initial charter
  under consideration. Fixes #2458.

• Made sure that sessions listed in session scheduling notification
  emails are listed in time order. Fixes issue #2460.

• Added TeX escaping utility functions and template filters. Removed
  html escaping and added TeX escaping for relevant parts of the bibtext
  template. Fixes issue #2459.

• Removed an option which could disagree with memcached.

• Updated PLAN

Coverage

Summary: Django 1.11
Release Date: Mon, February 26, 2018 at 3:03 PM UTC
Release Author: Henrik Levkowetz

This release upgrades the datatracker to use version 1.11 of the Django
framework. Django 1.11 is designated as a long-term support release. It
will receive security updates at least until April 4, 2020. The next Django
release, 2.0, will require Python 3.4 or higher.

Django 1.11 does not have any really notable features; but it introduces
template-based widget rendering, deprecating the earlier class-based
approach to creating custom widgets, adds subquery expressions, class-based
model indexes, and some minor features, see [a].

A full diff of the upgrade changes is available in changeset [14695].

• Merged in ^/personal/henrik/6.72.1-django-1.11@14676: Upgrade to Django
  1.11

• Adjusted ordering for a number of meeting-related models to make the
  most relevant entries appear first in lists and drop-downs. Fixes issue
  #2457.

• Updated the requirement on xml2rfc version.

[a] https://docs.djangoproject.com/en/1.11/releases/1.11/

Coverage

Release Date: Fri, February 23, 2018 at 5:16 PM UTC
Release Author: Henrik Levkowetz

This release addresses a quite annoying bug that could occur when saving
reviewer assignments, and widens the short floor indication string that is
shown on the agenda from 2 to 3 characters. Also addresses a couple of
other bugs:

• Rewrote the head-of-rotation review assignment loop to guarantee that it
  will terminate. This should fix the gateway timeout issues we've seen
  lately, when review request volume in some review teams have been such
  that multiple assignments to the head of queue members have triggered the
  issue.

• Changed the floorplan short string from 2 to 3 characters, in order to
  support things like London Hilton Metropole's level -3E, 3rd lower ground
  floor East.

• Changed the arguments to the xml2rfc parser to use normalize=True, in
  alignment with xml2rfc internal use.

• Changed an exception handling to not depend on exceptions having a .msg
  element.

Coverage

Release Date: Thu, February 22, 2018 at 6:33 PM UTC
Release Author: Henrik Levkowetz

This release provides a bugfix and updates a requirement to avoid versions
of the bibtexparser that don't work under Python 2.7 or require changed
parser initialization, cleaning the slate for the Django 1.11 upgrade.

• Merged in [14655] from housley@vigilsec.com:
  Strip the HTML tags around the new values in the mail produced by
  email_iesg_processing_document. Also, provide the previously missing URL
  at the end of the message. Fixes #2455.

• Added code to catch IOError exceptions when trying to apply patches.

• Updated version requirement for bibtexparser.

Coverage

Release Date: Fri, February 16, 2018 at 2:46 PM UTC
Release Author: Henrik Levkowetz

Bugfixes and minor features:

• Added page counts for open review requests on the reviewer assignment
  drop-downs of /group/*/reviews/manage/unassigned/. Fixes issue #2449.

• Added a patch for a faker problem with locale ro_RO names (default
  first_names from a region without first_names explicitly declared are
  returned as str instead of unicode under python 2.7,
  joke2k/faker#684)

• Added a presence check before removing a list element. Addresses a 500
  in unassigned review form POST.

• Fixed an issue which could arise when updating IPR declarations with no
  value for patent_info.

• The IPR disclosure form field for statement is too small to hold some
  of the existing statements. Increased from 255 to 2000.

• Added a missing indirect dependency on PyOpenSSL (via urllib3) to
  requirements.txt, to work around a broken dependency chain.

• Merged in [14634] from rjsparks@nostrum.com:
  Simplified the view that lets the secretariat see and change timeslot
  types. Fixes #2313.

• Updated the LICENSE file to used the wording from
  https://opensource.org/licenses/BSD-3-Clause

Coverage