Summary: Nomcom, agenda, and test improvements, new API for secretariat use
Release Date: Mon, October 15, 2018 at 5:46 AM UTC
Release Author: Henrik Levkowetz

This is a small feature release, with the following changes:

• Updated the nomcom admin pages for Feedback to make it possible to see
  and sort on the Nominees field.

• Merged in [15551] from rjsparks@nostrum.com:
  Used factories instead of make_test_data for malinglists,
  mailtrigger, message, person, secr/announcement, and secr/areas tests.

• Merged in [15550] from rjsparks@nostrum.com:
  Added meeting.agenda_info_note.

• Merged in [15549] from rjsparks@nostrum.com:
  Renamed meeting.agenda_note to meeting.agenda_warning_note.

• Changed the View button on the NomCom email classification page to
  render the email body within

  , and to not offer classification
  
  choices, as this encourages a slow workflow, and also seems to have been
  
  broken for some time.
  
  

• Bleach 3.0.0 has a bug that can result in exceptions for some inputs.
  Bug report filed. Disallowed that version in requirements.txt

• Turned the nomcom chair dropdown menu item Classify pending feedback
  into a tab with a mail count badge.

• Merged in [15263], [15264], [15265], [15287] from rcross@amsl.com:
  Added new custom API endpoints for export of person data, restricted
  to secretariat use. Added a test for the new custom API.

Coverage

Summary: NomCom tool improvements
Release Date: Mon, October 8, 2018 at 1:23 PM UTC
Release Author: Henrik Levkowetz

This is a feature release with only small tweaks, mostly related to the
NomCom tools, but there are also some IESG page, test suite, account, and
utility script improvements,

• Added pagination to the nomcom pending-feedback page, to reduce load
  time.

• Changed the default sort for nomcom.models.Nominees to be
  person__name.

• Merged in [15523] from rjsparks@nostrum.com:
  Converted the liaisons tests to use factories.

• Added a --reminder switch to the GDPR request mailer, for repeat
  notifications.

• Merged in [15522] from rcross@amsl.com:
  Made email address read-only in rolodex edit form. Fixes #2602.

• Prevent issues for logins without Person records by not letting
  Person-less users through login.

• Added an send_email field to BallotPostitionDocEvent to reflect whether
  the Send email button was used to save the ballot position. Added icons
  on documents' IESG evaluation record to reflect this.

• Adjusted the test_submit_nonascii_name() to match the output from
  idnits 2.16.0, which emits warnings rather than errors for nonascii content.

• Built out test_edit_nomcom_view() to also test adding and removing
  reminder dates.

• Added a form clean() for the reminder dates of the nomcom edit form, to
  make something appropriate out of edits that remove reminder dates, rather
  than clicking delete. This fixes a potential exception, when a blank
  date cannot be saved, and also does what the user expected (deletes a
  blanked date).

• The nomcom send_reminders command was not being run daily. Fixed that.

• Fixed a typo in the GDPR consent request email template.

• Merged in [15492] from rjsparks@nostrum.com:
  Use factories for ipr tests.

Coverage

Summary: Email address managment and NomCom tool improvements
Release Date: Sun, September 30, 2018 at 2:47 PM UTC
Release Author: Henrik Levkowetz

This release brings no new user features, but provides a script that
will read email delivery-status messages and disable addresses with
permanent delivery failures. There are also some other tweaks:

• Added a management command to deactivate email addresses found in
  delivery-status emails or given on the command-line.

• Added column sorting to the private nomcom nominee table.

• Tweaked the management command to send GDPR consent request emails, to
  correctly perform exclusion based on the time of previous consent request
  emails.

• Added a PersonEvent type for email address deactivation.

• Tweaked PersonAdmin to show name_from_draft after the name field.

• Added a clarification to the GDPR consent request email text after
  receiving about 100 reply emails in response to the first email round that
  contained what seems to be automatic responses indicating 'Yes, I
  confirm'. All those were from gmail addresses. The consent request
  now clearly says that people need to log in to their account in order
  to indicate consent.

Coverage

Summary: Improvements in NomCom pages and personal information handling
Release Date: Fri, September 28, 2018 at 11:50 PM UTC
Release Author: Henrik Levkowetz

This is a combined feature and bugfix release, with fixes related
to User and Person record handling and merging, discovered when people
responded to the GDPR consent request emails, and enhancements in
the NomCom tools:

• Added username validation to the change username form, to avoid
  database key collision exceptions. Instead give a message to contact the
  secretariat to resolve the situation.

• Fixed a problem where an exception could be raised when reporting an
  exception during draft XML processing to the user.

• Added consent to the fields handled when merging person records.

• Merged in [15486] from rjsparks@nostrum.com:
  Allow a nomcom to show nominees that have accepted nomination before
  feedback is open. Fixes #2598.

• Tweaked the consent request email text to point out an alternative to
  follow email links.

• Merged in [15450] from rcross@amsl.com:
  Change content-type of ajax_get_liaison_info to application/json.

• Tweaked the text of the personal information help page to remove some
  ambiguity.

• A string.Template with a unicode template must be given unicode
  dictionary values if the values contain unicode codepoints. Providing
  values that are objects with str() methods, or utf-8 encoded strings
  doesn't work. Fixes an issue with nomcom template interpolation for
  nominees with non-ascii names.

• Tweaked the GDPR reminder command to avoid bounces to RT and copies to
  gmail.

• Changed the sort order of nominees on the private nomcom index page to be
  according to name (instead of email address).

• Fixed an issue where a user record without matching person record will
  cause an exception.

• Removed the 2.5 hour session length choice for IETF 103, since there
  will be no Friday sessions that meeting.

• Updated the personal information notice email text based on advice from
  councel (Tom Zych).

Coverage

Summary: GDPR updates part 4
Release Date: Tue, September 18, 2018 at 2:07 PM UTC
Release Author: Henrik Levkowetz

This is the last in a series of GDPR-related releases. It introduces two
management commands; one to email out reminders to all users that still have
not provided consent for storage of the personal information for which the
GDPR requires consent, and the other to delete all information for which
we don't have required consent.

The two new management commands are:

- ietf/manage.py send_gdpr_consent_request
- ietf/manage.py delete_data_lacking_consent

Both of these provide descriptions of the commands and switches when given
a --help switch, to assist the secretariat in using these.

There are also a few bugfixes and features not related to the GDPR, see
below:

• Merged in the latest GDPR changes. This refines the handling of the
  consent checkbox on the account page; refines the Consent Needed warning
  given on login if consent is needed; tweaks several models to set the
  on_deletion fields for FK to User and Person appropriately; adds a
  Person.needs_consent() method to capture the logic of which fields require
  consent; refines the Person.plain_name() method and the user.log.log()
  function; and adds 2 management commands to send out consent requests and
  delete non-consent information, respectively.

• Refined utils.log.log() to flush stdout before logging if we're writing
  log messages to stderr.

• Moved document methods rfc_number() and rfcnum() to the abstract parent
  class to have them available also for historical document records. Should
  fix the recent server 500 on /doc/draft-ietf-aaa-diameter/17/.

• Merged in [15448] from rjsparks@nostrum.com:
  Add CORS headers for the api and views that serve json. Fixes #2546,
  #2551, and #2552.

• Added a bit of javascript and styling to make it possible to use URL
  fragment identifiers to link to position-specific tabs on the nomocom
  requirements and questionnaire pages.

• Merged in [15446] from rjsparks@nostrum.com:
  Use unversioned links in ics agendas. Partially fixes #2542.

• Merged in [15444] from rjsparks@nostrum.com:
  Review assignment email includes a link to the review request that
  ensures the reviewer is logged in. Fixes #2543.

• Added a Mailing list discussion link to the Additional URLs section
  of the status page for drafts with doc.group and doc.group.list_archive set.

• Merged in [15441] from peter@akayla.com:
  Fixed 2 trivial spelling errors in NomCom tools. Fixes issue #2569.

• Fixed a problem with fetching the mysql release signing key from a gpg
  keyserver.

• Added more hint information when patch actions in ietf/checks.py fail.

• Rewrote docker/run to work around a missing switch in BSD xargs.

• Merged in [15435] from rjsparks@nostrum.com:
  Include area groups in the request for minutes. Do not include sessions
  that have been canceled. Fixes #2568.

Coverage

Release Date: Tue, August 14, 2018 at 12:31 PM UTC
Release Author: Henrik Levkowetz

This is a small bugfix and cleanup release:

• Replaced some cases of using person.plain_name when linking to a person's
  profile with person.name, as the link using plain_name can fail.

• Generalized the log.assertion() function a bit to be able to provide
  more information on assertion failure.

• Merged in [15423] from rcross@amsl.com:
  Removed WG milestone management from secretariat groups app.

• Removed youtube video url import, which is not needed any more given the
  /api/meeting/session/video/url API endpoint. This also prevents duplicate
  video URLs.

• Changed the menu listing of review dirs and their display on the
  Directorates page back to what it was before the reviews group type was
  split out. Added a Reviews column to the Directoratges page.

• Added a guard against comparing email origin with username of None.

• Added a None guard for the person merge last_login date comparison.
  Fixes issue #2562

Coverage

Summary: IETF 102 code sprint
Release Date: Wed, July 25, 2018 at 7:08 PM UTC
Release Author: Henrik Levkowetz

This release contains datatracker bugfixes and enhancements from the
IETF-102 Code Sprint. While we've had many more code sprint participants
before, I don't believe we've ever had this number of contributions from an
IETF week :-)

The release contains a number of different enhancements to the document
review pages (thanks Tero, Peter); refactoring of a number of tests to use
test object factories instead of indiscriminate creation of a large number
of often unused test objects, thus speeding up the tests (thanks to Robert);
fixes to template language and anchors (thanks to Jean); a fix for annoying
issue with a liaison form (thanks, Ben); a number of updates to secretariat
pages and proceedings (thanks, Ryan); a fix for a long-standing issue with
the document search forms behaviour when no search string has been given
(thanks, Jim); and a test to expose bugs in downref listing in last call
announcements (thanks, Russ).

From the commit log:

• Show links to personal profile pages from group photo pages also when no
  photo is available -- the profile pages now contains more than just the
  photo.

• Merged in [15399] from rjsparks@nostrum.com:
  Use factories for ietf.ietfauth.tests.

• Merged in [15395] from rjsparks@nostrum.com:
  Allow the IRTF stream to set shepherds and shepherd writeups. Fixes #1993.

• Merged in [15387] from rjsparks@nostrum.com:
  Use factories for ietf.iesg.tests.

• Merged in [15380] from rjsparks@nostrum.com:
  Use factories for ietf.idindex.tests.

• Merged in [15370] from rjsparks@nostrum.com:
  Use factories for ietf.group.tests_review.

• Merged in [15358] from kivinen@iki.fi:
  Fixed the review tablesorter issue by adding display:none character in the
  front of the date. Tablesorter looks first characters and picks the parser
  based on that, meaning that if the text starts with number, it will try to
  parse everything in column as numbers, and if there is anything that is
  not numebr it will fail, and only sorts the column once. This is bit ugly
  but I do not think we want to add yet another jquery meta tool. Fixes
  #2089, and #2151.

• Added a missing migration (for review unavailable-period reason)

• Merged in [15357] from mahoney@nostrum.com:
  Added anchors for working group rows. Still needs some work since the
  anchor ends up being under the menu bar. Partially addresses issue #1481.

• Merged in [15356] from ben@nostrum.com:
  Changed newlines to commas in to contacts for new liaison email
  notification. Fixes issue 2525.

• Merged in [15355] from rjsparks@nostrum.com:
  Used factories for ietf.group.tests_info.

• Tweaked a test to prevent failures when looking for a specific string
  in line-wrapped text.

• Added a tiny utility function unwrap() to unwrap wrapped text for
  matching expected strings in tests.

• Merged in [15354] from kivinen@iki.fi:
  Added default to review completion draft version number. If the reviewer
  is filling in the review form himself the default will be the latest
  version of the draft. If it is someone else filling in the review
  then the version number is the latest version at the point when review
  was assigned (i.e., it assumes reviewer did review on version he was
  requested to and then document authors might have updated the document).
  Also mark the document versions that are earlier than the review request
  date with red background and those which are newer with green background.
  Fixes #2125.

• Merged in [15353] from peter@akayla.com:
  Add a sidebar to display feedback section names and simplify navigation.
  Could use some visual improvements but otherwise works. Fixes #2431.

• Merged in [15351] from rcross@amsl.com:
  Update secr/drafts/tests_views to align with view logic. Fixes #2480.

• Merged in [15349] from kivinen@iki.fi:
  Removed the requirement for review_content field to be mandatory when
  using 'link' method. This partially addresses #2520.

• Merged in [15347] from kivinen@iki.fi:
  Add review request date to the top. Add tooltips for drafts versions
  providing the date they were sumbmitted. Fixes #2218.

• Limited the document name for uploaded slides to use 128 slugified
  characters from the slideset title as part of the name. Fixes a possible
  server 500 issue.

• Merged in [15345] from rcross@amsl.com:
  Remove duplicate functionality in secr/drafts. Fixes #2540.

• Merged in [15344] from mahoney@nostrum.com:
  email confirming interim meeting request now says 'the Area Director of
  the ' instead of <area> Area Director. Fixes issue #2522.

• Added additional checks to the meeting_session_video_url api test.

• Fixed api_set_session_video_url() to update the external video url if
  given a new value.

• Merged in [15342] from peter@akayla.com:
  Added optional reviewer unavailability reason support. Fixes issue #2146.

• Merged in [15335] from rcross@amsl.com:
  Fixed the ordering of meeting select in secr/meetings. Fixes #2539.

• Adjusted some timezones in test code to use permitted time zones after
  the previous changes in listed timezones. Added a migration for the
  changed timezone list.

• Fixed display of xml2rfc exceptions when processing draft submission in
  xml form.

• Modified the list of timezones available when setting a meeting
  timezone so as to not show timezones that are links. (Also edited the 102
  meeting timezone in the database to avoid this). Fixes issues #2531 and
  #2534.

• Merged in [15338] from fenton@bluepopcorn.net:
  Adds required parameter to search widget. Fixes #1832.

• Merged in [15331] from housley@vigilsec.com:
  Add test for downref in Last Call announcement

• Merged in [15329] from rcross@amsl.com:
  Add important dates link to proceedings. Fixes #2538.

• Merged in [15328] from rcross@amsl.com:
  Give secretariat the ability to cancel non-working group sessions.
  Fixes #2537.

• Updated a test to give predictable results (no future meeting dates)
  for commit [15361].

• Fixed an issue where /meeting/ showed the materials page of an earlier
  meeting, rather than the current.

• Merged in [15327] from mahoney@nostrum.com:
  Clarified description in the search form, removing the etc since you
  can't search on author names. Fixes issue #1159.

• Merged in [15321] from rjsparks@nostrum.com:
  Converted ietf.group.tests to factories.

• Merged in [15320] from rjsparks@nostrum.com:
  The cookie tests were all calling make_test_data, but not using anything
  it created. Removed the calls.

• Merged in [15314] from rjsparks@nostrum.com:
  Converted the community app tests to use factories.

Coverage

Summary: Group features moved from code to database
Release Date: Sat, July 14, 2018 at 4:47 PM UTC
Release Author: Henrik Levkowetz

This release replaces the code defined group features with features held
in a database table:

• Added a GroupFeatures model to the group models, and removed the old
  features.py
• Added a agenda type for future use in showing different group types on
  different agendas.
• Renamed the group feature has_materials to has_nonsession_materials.
• Added API resources and admin support for the new tables.
• Added a Directorate (with reviews) group type as complement to
  Directorate, to distinguish between directorates with and without
  reviews.
• Adjusted tests as needed.
• Updated the fixtures, and fixed the generate_fixtures script to include
  the new AgendaTypeName objects.

There still exists about 70 instances of code comparing the group type
with a list of types; most of these should probably be replaced with new
features, instead, to make it possible to add new group types through the
database table, rather than having to edit the code. That was the purpose
of this refactoring from the start, but the presence of this large number
of comparisons of group type against lists of types defeats the goal until
we add appropriate features and replace the group type list comparisons.

Additional changes:

• Added exception logging for ietf/bin/rfc-editor-queue-updates. Changed
  the logging in the ietf/bin/rfc-editor-* scripts to use
  ietf.utils.log.log(). Reordered some imports.

• Replaced the use of a deprecated Django field attribute with the
  sanctioned replacement.

• Merged in [15282] from housley@vigilsec.com:
  Set reply-to for I-D Announcements that include a CC to a mail list.
  Fixes #2530.

• Added support for patent kind codes as part of patent numbers.

• Tweaked the rsync arguments for IANA YANG models.

• Upped the permitted number of search results for the /stream/ise/ page
  (and friends) from 200 to 1000. Fixes issue #2527.

Coverage

Release Date: Wed, June 20, 2018 at 9:47 PM UTC
Release Author: Henrik Levkowetz

This is a bugfix release to clear the board for the final GDPR release.
From the commit log:

• Merged in [15266] from rcross@amsl.com:
  Updated the blue sheet template. Replaced start time / end time with date.

• Merged in [15267] from rcross@amsl.com:
  Fixed an issue with a decorator on utils.html.remove_tags().

• Tweaked the Related documents part of the Document admin page to more
  intuitively show the relationships.

• Tweaked unidecode_name() to avoid double-quotes in the ascii string.

• Merged in [15248] from rjsparks@nostrum.com:
  Improvements to tests after converting to factories suggested by studying
  coverage_changes.

• Merged in [15247] from rjsparks@nostrum.com:
  Tweaked the color palette for the agenda editor.

• Removed two instances of log.assertion(); we have enough data on these
  now.

• Changed an attribute guard to work with a reverse relationship attribute
  (using hasattr()).

• Fixed a broken mailto link on the account edit page.

• Merged in [15238] from rjsparks@nostrum.com:
  Use factories for ietf.doc.tests_status_change. Repair previous commit
  errors in ietf.doc.tests_material.

• Merged in [15228] from rjsparks@nostrum.com:
  Use factories for ietf.doc.tests_downref.

• Merged in [15227] from rjsparks@nostrum.com:
  Use factories for ietf.doc.tests_conflict_review.

• Merged in [15226] from rjsparks@nostrum.com:
  Use factories for ietf.doc.tests_charter.

• Tweaked unidecode_name() to not produce single-letter ascii surnames
  from non-ascii codepoints. The unidecode transliteration is in any case
  somewhat arbitrary, and in most cases a real person will tweak the ascii
  name of his account. When running tests, however, this tweak avoids some
  false test failures. And no, it's not simple to fix the draft
  author-extraction heuristics to deal well with single-letter surnames.

• Added a guard against trying to access attributes of a None object.

• Merged in [15225] from rjsparks@nostrum.com:
  Use factories for ietf.doc.tests_draft.

Coverage

Summary: GDPR updates part 3
Release Date: Fri, June 8, 2018 at 4:05 PM UTC
Release Author: Henrik Levkowetz

This is the third in a small series of releases to address compliance with
the European GDPR (General Data Protection Regulation). It provides
serialized (machine-readable) access to information associated with a user's
account, adds photo and some links to the user account page, and improves
the annotation of which information is consent-based on the account page.
It adds a warning on login if consent is missing for some personal
information, and requests the user to give consent. It also brings in
refactoring of a number of test cases to use test object factories, giving
improved test performance.

• Merged in [15219] from rjsparks@nostrum.com:
  Use factories for ietf.doc.tests_ballots. Fixed a long-hidden bug in
  test_edit_verify_permissions where non-existing usernames were used and
  passed the tests accidently. Added list_email to the GroupFactory.

• Merged in [15217], [15205], [15204], [15203], [15202], and [15201] from
  rjsparks@nostrum.com:
  Converted ietf.doc.tests to use object Factories. Created draft type
  specific factories.

• Fixed the logic of gathering which personal information needs consent
  confirmation in the code that requests consent on login.

• Added a request to give consent for non-LI information on user login.
  Addresses issue #2505.

• Fixed a bug in a form clean method.

• Added a missing related object to the person information export.
  Related to issue #2501.

• Tweaked the language on the user account page. Related to issue #2501.

• Added a recursive object to JSON serializer and a view which will let
  any logged-in user download a JSON serialized copy of the datatracker
  information related to his person record. Added information about this,
  and a link, to the account page. Related to issue #2501.

• Added a requirement of oauth2client, needed by google-api-python-client
  but not always automatically pulled in.

• Added a link to the overall IETF Statement concerning personal data to
  the datatracker personal-information page.

• Added some text about BOF requests to the important-dates page for the
  session request opens dates, on request from the IESG.

Coverage