cmtools/doc/xscore_gen.md

xscore_gen
==========

*xscore_gen* parses MusicXML score files and generates a text file
which allows the score to be clarified and additional information to
be added.  This is the first step in creating a 'machine readable
score' based on the 'human readable score'.

This step is necessary because we need a way to efficiently 
append additional information to the score which cannot be entered
directly by the score editing program (e.g. Sibelius, Dorico, Finale).
In practice we use Sibelius 6 as our primary score editor.

Likewise there are certain limitations to the generated MusicXML
which need to be worked around. The primary problem being that
dynamic markings are not tied to specific notes. This is important
for purposes of score analysis as well as audio rendering. 

The overall approach to adding this addtional information
is as follows:

1. Add as much auxilliary information as possible from within Sibelius.
This entails using colored note heads, and carefully placed
text strings.

2. Generate the MusicXML file using the [Dolet 6 Sibelius
plug-in](https://www.musicxml.com/).  The resulting MusicXML file is
run through *xscore_gen* and parsed to find any invalid structures
such as damper up events not preceeded by damper down events, or tied
notes with no ending note. These problems are cleared by careful
re-editing of the score within Sibelius until all the problematic
structures are fixed.

3. As a side effect of step 2 a template 'decoration' file is generated.
This text file has all the relavant 'machine score' information 
from the XML score as a time tagged list. In this step *decoration* information is manually
added by entering codes at the end of each line.  The codes
are cryptic but they are also succinct and allow for relatively
painless editing.

4. Once the addition information is entered *xscore_gen* is
run again to generate three output files: 

- machine score as a CSV file
- MIDI file suitable for audio rendering
- SVG piano roll file

As with step 2 this step may need to be iterated several times
to clear syntactic errors in the decoration data.

5. Generate the time line marker information to be used with the performance program:

Generate the time line marker information, into `temp/time_line_temp.txt` like this:

`cmtest -F`

This calls `cmMidiScoreFollowMain()` in app\cmMidiScoreFollow.c.

Then paste `temp\time_line_temp.txt` into kc/src/kc/data/round1.js.



Preparing the score
-------------------

Note color is used to assign notes to groups. 
These groups may later be used to indicate
certain processes which will be performed
on these notes during performance.

There are currently three defined groups
'even','dynamics' and 'tempo'.


### Score Coloring Chart:

Description         Number   Color
------------------- -------- -------------------------
Even                #0000FF  blue 
Tempo               #00FF00  green
Dyn                 #FF0000  red
Tempo + Even        #00FFFF  green + blue (turquoise)
Dyn   + Even        #FF00FF  red   + blue
Dyn   + Tempo       #FF7F00  red   + green (brown)
Tempo + Even + Dyn  #996633  purple

Decrement color by one (i.e. 0xFE) to indicate the last note in a group
of measured notes.  Note that a decremented color stops all active measures
not just the measurement associated with the decremented color.  


Preparing the Music XML File
----------------------------

*xscore_gen* is know to work with the MusicXML files produced by
the [Dolet 6 Sibelius plug-in]<https://www.musicxml.com/>

After generating the file it is necessary to do some
minor pre-processing before submitting it to *xscore_gen*

iconv -f UTF-16 -t UTF-8 -o score-utf16.xml score-utf8.xml


Create the decoration file
--------------------------

```
cmtools --score_gen -x myscore.xml -d mydec.txt
```

Here's a snippet of a typical 'decoration' file.

```
Part:P1
  1 : div:768 beat:4 beat-type:4 (3072)
      idx voc  loc    tick  durtn rval        flags
      --- --- ----- ------- ----- ---- --- ---------------
        0   0     2       0     0  0.0     |--------------
        1   0     0       0    54  4.0     --------------- 54 bpm
        2   1     0       0  3072  1.0     -R-------------
        3   5     0       0  2304  2.0     -R-.-----------
        4   0     0     996     0  0.0     --------V------
        5   0     0    1920     0  0.0     --------^------
        6   5     0    2304   341  8.0     -R-------------
        7   0     0    2643     0  0.0     --------V------
        8   5     0    2645    85 32.0     -R-------------
        9   5     3    2730    85 32.0 F 5 --------------*
       10   5     4    2815    85 32.0 Ab2 --------------*
       11   5     5    2900    85 32.0 C 3 --------------*
       12   5     6    2985    87 32.0 F 6 --------------*
       13   1     0    3072   768  4.0     -R-------------
       14   5     0    3072   768  4.0     -R------------- 3840
```

### Decoration file format

Column | Description
-------|-----------------------------
idx    | event index
voc    | voice index
tick   | MIDI tick 
durtn  | duration in MIDI ticks
rval   | rythmic value
pitch  | scientific pitch
flags  | event attributes 

### Event attribute flags:

Event attribute symbols used in the decoration file:

Desc      | Flag | 
----------|------|-----------------------------------------
Bar       |  |   | Beginning of a measure
Rest      |  R   | Rest event
Grace     |  G   | Grace note event
Dot       |  .   | note is dotted
Chord     |  C   | note is part of a chord
Even      |  e   | note is part of an 'even' group
Dyn       |  d   | note is part of a 'dynamics' group
Tempo     |  t   | note is part of a 'tempo' group
DampDn    |  V   | damper down event
DampUp    |  ^   | damper up event
DampUpDn  |  X   | damper up/down event
SostDn    |  {   | sostenuto down event
Section   |  S   | section boundary
SostUp    |  }   | sostenuto up event
Heel      |  H   | heel event 
Tie Begin |  T   | begin of a tied note
Tie End   |  _   | end of a tied note
Onset     |  *   | note onset  



Decoration Sytax:
------------------

!<dyn_mark>             Assign dynamics
!(<dyn_mark>)             - less uncertain dynamic mark
!<upper-case-dyn-mark>    - begin of dynamic fork (See note below regarding dynamic forks)
!!<upper-case-dyn-mark>   - end of dynamic fork                      
~<mark>                 Insert or remove event (See pedal marks below.)
@<new_tick_value>       Move event to a new time position
%<grace_note_flag>      Flag note as a grace note
%%<grace_note_flag>         -last note in grace note sequence
$<sci_pitch>            Assign a note a new pitch

<grace_note_flag>
  b (begin grace)
  a (add grace and end grace)
  s (subtract grace and end grace)
  g (grace note) 
  A (after first)
  N (soon after first)

Note: The first non-grace note in a grace note sequence is marked with a %b.
The last non-grace note in a grace note sequence is marked with a %s or %a.

Where: %s = steal time from the note marked with %b.
       %a = insert time prior to the note marked with %a.
       
 The last (by row number) note (grace or non-grace) in the the sequence
is marked with %%# where # is replaced with a,b,s,or g.

It is only necessary to mark the tick number of grace notes in order
to give the time sequence of the notes. A single grace note therefore does
not require an explict tick mark notation (i.e. @####)


Insert/delete  Event Marks: <mark>
-----------------------------------
 d (sostenuto down - just after note onset)
 u (    "      up  - just before this event)
 x (    "      up    just before this event and down just after it)
 D (damper pedal down - after this event)
 U (damper pedal up - before this event)
 _ (set tie end flag)
 & (skip this event)

Dynamic Marks: <dyn-mark>
--------------------------
  s       (silent note)
  pppp-
  pppp
  pppp+
  ppp-
  ppp
  ppp+
  pp-
  pp
  pp+
  p-
  p
  p+
  mp-
  mp
  mp+
  mf-
  mf
  mf+
  f-
  f
  f+
  ff
  ff+
  fff

Note: Dynamic Forks:
--------------------
Use upper case dynamic letters to indicate forks in the dynamics
which should be filled automatically.  Note that all notes
in the voice assigned to the first note in the fork will be
included in the dynamic change. To exclude a note from the
fork assign it a lower case mark.