.. Documentation of matholymp static site generation. Copyright 2014-2020 Joseph Samuel Myers. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Additional permission under GNU GPL version 3 section 7: If you modify this program, or any covered work, by linking or combining it with the OpenSSL project's OpenSSL library (or a modified version of that library), containing parts covered by the terms of the OpenSSL or SSLeay licenses, the licensors of this program grant you additional permission to convey the resulting work. Corresponding Source for a non-source form of such a combination shall include the source code for the parts of OpenSSL used as well as that of the covered work. .. _static-site: Static site generation ====================== Matholymp supports generating a static site with information about past competitions. This is done with the :command:`mo-static-generate` script. Before using this script, you need to set up the directory that will contain the site. This will contain files :file:`staticsite.cfg` and :file:`page-template`, as well as a directory :file:`data/` with the information about past events. Some parts of the site are not generated by this script and need to be set up manually. :command:`mo-static-generate` currently expects there to have been at least one past event. Thus, if setting up a web presence for a new competition, you should defer running :command:`mo-static-generate` until after the first event (although other parts of the site can still be set up). :command:`mo-static-generate` should be run from the directory containing the site. It reads the files listed above and generates other files (overwriting them if they already exist). It does not delete files even if they were generated by a previous run of :command:`mo-static-generate` and are no longer generated because of changed data (in such cases, e.g., if the record for a person was removed because they were not in fact at the event they were listed at, the files must be removed manually). Example files ------------- The :file:`examples/static-site/` directory in the matholymp source distribution includes versions of :download:`staticsite.cfg <../examples/static-site/staticsite.cfg>` and :download:`page-template <../examples/static-site/page-template>` that may be used as a basis for configuring the generation of your site. The example shows a site accessed over HTTPS, since this is a good idea for the registration system, although all the data on the static site itself is public. It is expected that if both the static site and the registration system are used, the registration system uses URLs of the form :samp:`registration/{year}/` relative to the top of the static site. Site design ----------- It is up to you to determine the visual design of your site and to set up an appropriate HTML template, using appropriate CSS styles, that implements this visual design. If you wish, you can generate :file:`.html` files directly with :command:`mo-static-generate` and put your complete HTML template directly in the :file:`page-template` file. Alternatively, you may wish to use some external templating system to apply your design both to the files generated by :command:`mo-static-generate` and to the manually maintained parts of the site. You could, for example, configure :command:`mo-static-generate` to create :file:`.php` files (using ``page_suffix = .php`` in :file:`staticsite.cfg`), with :file:`page-template` containing the PHP boilerplate to include the bulk of the page template from other files. Or you could post-process the files generated by :command:`mo-static-generate` rather than using them directly on the website at all. You may wish to put extra manually-maintained content on event pages, such as links to regulations or solutions. To support this, :file:`staticsite.cfg` specifies ``page_include_extra`` as text that goes on event pages to include such extra content; ``%(dir)s`` in that text is replaced by the name of the directory for that event. For this to be useful, you need some templating system (possibly as minimal as server-side includes, as shown in the example file) that can include such extra content when encountering such text. Similarly, ``scoreboard_include_extra`` is text that goes on scoreboard pages; ``%(dir)s`` in that text is replaced by the name of the directory for the event (not for the scoreboard itself). Your page template should reference a stylesheet which provides styles whose names are specified in :file:`staticsite.cfg` for various constructs in the generated pages. The `corresponding styles used for EGMO `_ may give a useful idea of some possibilities, although of course you will need to make your styles fit in with your overall site design. Your page template, or a manually maintained page, should contain links to the overall ``countries/``, :samp:`{event}s/` and ``people/`` pages on the site. You also need to create a front page to the site manually. Miscellaneous generated files ----------------------------- :command:`mo-static-generate` generates some files that you may wish to use in your site, but are not used automatically. These are generated in the :file:`{event}/auto/` directory. * :file:`{event}-contact-list{suffix}` is an HTML fragment with contact details for each event, which you may wish to include in some manually-maintained page. * :file:`sidebar-{event}-list{suffix}` is an HTML fragment listing events in a form that may be suitable to include in a sidebar in your page template. * :file:`redirects-{n}` is a file listing redirection rules to redirect accesses to the registration system for event *n* to the corresponding pages on the static site, so that public URLs for registration system pages continue to work after the registration system is shut down. These rules are in the format used by `Apache mod_rewrite `_, which must thus be enabled in your Apache configuration if you wish to use such redirects. You will also need to set up Apache to use these rules. Data files ---------- :command:`mo-static-generate` generates the site using data files in the :file:`data/` directory. These are UTF-8 CSV files starting with a byte order mark. The files for EGMO (`egmos.csv `_, `papers.csv `_, `countries.csv `_, `people.csv `_) may be useful in illustrating the format. The first file has a name that depends on ``short_name_url_plural`` in :file:`staticsite.cfg`; the names of the other files do not depend on the event. The names of some columns in these files depend on the settings of ``num_key`` and ``official_desc``. The file :file:`{event}s.csv` has to be maintained manually; this means entering details of past events, entering details of new events when announced and putting in medal boundaries after each event. The other files can be updated during and after each event by the scripts :command:`mo-static-papers-import` and :command:`mo-static-import`, as described below. Some columns in :file:`{event}s.csv` are optional, and are not present in the example for EGMO but may be added if certain information varies from event to event so cannot go in :file:`staticsite.cfg`. A column ``Honourable Mentions Available`` may be used to specify whether the rules of a particular event allowed for Honourable Mentions to be awarded; a column ``Distinguish Official Countries`` may be used to indicate whether there was a distinction between official and unofficial countries that should be reflected in the lists of countries and results. The ``Age Day Description`` column, present for EGMO, is also optional. Creating the other files initially, with details of past events, is more complicated. You may start with files simply containing the CSV header row (with BOM). That will suffice to add data for new events using :command:`mo-static-papers-import` and :command:`mo-static-import`. To add data for past events, you need to put it into the correct format for these CSV files (which includes identifying when the same person or country was at more than one event and using that to determine whether to assign new Person Number or Country Number values); you also need to copy papers and any photos or flags into the locations given in the relevant CSV columns. Because this depends on the form in which you have the data for past events, there is no general solution to this. You might have spreadsheets with data, in which case it may be appropriate to convert those to CSV format and then write your own Python script to convert that CSV file into exactly the right format. You might have web pages that need screen scraping to extract the data from tables on those pages. For older events you might have data on paper that needs scanning and careful proof-reading (in such a case, it would be a good idea to make the scans available on the site as well, in case any questions arise about possible mistakes). It may well be convenient, when adding data for past events, to put it in the format expected as input to :command:`mo-static-import` and then use that script to deal with assigning person or country numbers. This input format is almost the same as the format of the :file:`countries.csv` and :file:`people.csv` files on the static site; the script (:download:`matholymp/scripts/mo_static_import.py <../matholymp/scripts/mo_static_import.py>`) should be examined for the details of how the formats differ (in particular, the different meanings of Person Number and Country Number in the input files). Special prizes are represented in :file:`people.csv` by putting ``Special Prize`` in the ``Extra Awards`` column. If different types of special prizes are distinguished, this column may give a name or description of the prize (in general, it can contain a comma-separated list of extra awards, formatted as if the single row of a CSV file). .. note:: Matholymp does not currently support results of past events with partial information (such as total scores without scores on individual problems, or names not matched to scores). If you would like to put up such partial results, please contact me so we can work out how best to handle them. There is no problem, however, with putting up information where results of contestants are present but details of leaders and staff are missing, and then adding the details of those other people later when available. Static site maintenance tasks ----------------------------- The following describes how to carry out various maintenance on the static site. In all cases, the directory where you run the listed commands should be the top-level directory for the static site, containing :file:`staticsite.cfg`. Regeneration ^^^^^^^^^^^^ To regenerate the site (needed after any change to the configuration or data, including after following any of the instructions below for other tasks):: mo-static-generate When registration opens ^^^^^^^^^^^^^^^^^^^^^^^ Ensure that the relevant event is described in :file:`{event}s.csv`, and list the number of the event in ``event_active_number`` in :file:`staticsite.cfg`. Adding papers ^^^^^^^^^^^^^ To add a particular day's papers (day 1 in this example) to the site, where :command:`mo-document-generate` was used to produce papers with a logo on them, rather than papers being printed on paper with a pre-printed background design: .. parsed-literal:: mo-static-papers-import --day 1 *input-directory* where *input-directory* is the directory in which :command:`mo-document-generate` was run to generate the papers, with the :file:`data/` subdirectory containing its input CSV files and :file:`out/` containing the output PDF files. See :ref:`document-generation` for details of the use of :command:`mo-document-generate` to generate those files. To add a particular day's papers (day 2 in this example) to the site, where papers were printed on paper with a pre-printed background design, and both versions with and without that design are to be added to the site: .. parsed-literal:: mo-static-papers-import --background --day 2 *input-directory* In both cases, you may omit the :samp:`--day {day}` option if you are adding all papers at the same time, and must omit it if there is only one day at the competition. If the papers being added to the website have corrections relative to the versions used in the exam, you may wish to edit the Description column in :file:`papers.csv` to note this. Adding data after an event ^^^^^^^^^^^^^^^^^^^^^^^^^^ This should be done once the final scores and medal boundaries have been approved (and once any known corrections to participant names or other public details have been entered in the registration system). Ensure that the data for the relevant event in :file:`{event}s.csv` is complete, including the number of problems, the maximum number of marks for each problem and the medal boundaries. Choose a directory (*input-directory* below) outside the website for :command:`mo-static-import` to download files into. Make sure that ``event_active_number`` is set in :file:`staticsite.cfg`; that is required for :command:`mo-static-import` to download files automatically. Then: .. parsed-literal:: mo-static-import *input-directory* If this is the first event, at this point you should fill in Country Number for the host country in :file:`{event}s.csv` before regenerating the site (the number not having been allocated before then; this is the only reason it is not currently possible to use :command:`mo-static-generate` before the first event). If you wish to use the redirections from the registration system to the static site, you may add them to your Apache configuration after regenerating the site. Before doing this, make sure that anyone using the registration data administratively (e.g., to plan airport connections for departures) has all the data they need from the site. It is also possible to run :command:`mo-static-import` with files downloaded manually from the registration system rather than having it download them automatically. To do so, download the following files from the registration system (the lists of countries and people must be downloaded while not logged in administratively), and put them in the directory *input-directory*: the list of countries (:file:`countries.csv`), the list of people (:file:`people.csv`), the ZIP file of flags (:file:`flags.zip`), the ZIP file of participant photos (:file:`photos.zip`), the RSS feed of scores (save it as :file:`scores-rss.xml`). (The ZIP files do not need to be unpacked; if they are, the directories :file:`flags/` and :file:`photos/` will be used instead of the ZIP files.) Any files already present in that directory will be used by :command:`mo-static-import` in preference to downloading files automatically. Removing a photo ^^^^^^^^^^^^^^^^ If someone wants their photo removed from the static site, remove the files from the relevant directory (note that there may be multiple copies of it, for someone at multiple events, and thumbnail versions that need to be removed as well) and remove the entry or entries from the Photo URL column for that person in :file:`people.csv`, before regenerating the site. Removing a person or merging two people ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If it turns out that a person's record on the site (generally, or for a particular event) should be removed because that person was not present at the event they are listed at, or if there are multiple person numbers assigned to a single person at different events, then the relevant data needs editing directly. To remove a person, delete the problem rows from :file:`people.csv`. Remove any corresponding photos for the events that person was in fact not at. If the person was not at any event, remove their directory under :file:`people/`. Then regenerate the site. To merge two (or more) records for a person, decide which person number is to be canonical for that person. Change any rows in :file:`people.csv` with the noncanonical person numbers to use the canonical number. Move any photos from the directories with the noncanonical numbers to the directory with the canonical number, update the Photo URL columns accordingly and delete the noncanonical directory. Regenerate the site. If you wish, also add redirects in the web server configuration from the URLs of the removed directories to the canonical directory. .. note:: If the highest allocated person number is removed, a future use of :command:`mo-static-import` will reallocate that number to another person. You can avoid this by temporarily creating a dummy entry with that number in :file:`people.csv` before running :command:`mo-static-import`, then removing that entry afterwards. A future version of matholymp may provide a more automated way of handling removing and merging people.