Skip to content

Hark (surveys)

Status

This is a big project that is not documented well enough to be super accessible right now!

Create complex surveys. This tool is being developed for a particular purpose, but may have general use cases too.

Key features (mostly implemented!)

  • Multi page surveys can be defined with a typical set of data types (text, numbers) and input widgets (drop-downs, radios, checkboxes etc.)

  • Each participant gets a special access link. The design use-case is that the participant is an organisation and its staff will be completing the survey.

  • Safe simultaneous edits. Think locks on sections rather than realtime collaborative edits. Some sort of authentication/identification.

  • Embeddable in external website (implements Inlay)

  • Layered access: participants can see/edit a sub-set of questions that staff can see/edit. e.g. so a participant may be asked "how humble are you?" and may answer "I‘m probably the most humble person you’ve ever met", but then another question seen only by staff may ask "how humble do you think they are, score 1-5?" and staff may answer 1. Later permissions may be changed so that the participants can no longer edit their answers, but may still see them, and may also see the staff assessment of their answer.

  • Conditional questions; branches.

  • Sections, pages, groups.

  • Some sort of query/reporting/importing/exporting API and tools

  • Deadlines.

  • Edit logs.

  • phpunit tests

Key concepts/entities:

  • Surveys - the list of questions and metadata etc.

  • Participants - data for a particular survey. Participants are typically linked to an organisation.

  • Contributors - contacts who are allowed to contribute to a participant's answers in some role.

  • Role - a per-survey set of roles (simple surveys may just have a default 'admin' role) used to govern who can see/edit what data.

  • Part - Surveys are defined with lots of parts. A part may be a chunk of text, a question, the start or end of a section/page.

  • Answers - Belong to a Participant and a Survey.

See:

  • Other pages in this documentation, which may be better written!

  • Contacts » Hark Surveys (path: /civicrm/hark/surveys)

  • Inlay

  • Api4 explorer for Hark* entities

  • Contact Rich at Artful Robot if interested. (or @artfulrobot on chat)

Below may not be up-to-date and needs refactoring/rewriting. Probably better to read other pages than this one now


Survey structure.

This is a JSON object. It has a parts key which identifies an (ordered) array of parts.

{
  "parts": [ {PART}, ... ],
  "roles": [
    { name: "participant", description: "Unis" },
    { name: "assessor", description: "Staff assessors"}
    { name: "admins", description: "Staff admins", admin: true }
  ],
  "meta" : { ??? } }

Question structure.

  • id: short unique hash
  • codeName: short unique-within-survey codename, e.g. emr_co2_emissions
  • acl An object: { {ROLENAME}: "edit|view|use", ... }
    • edit means present as a question that can be edited. It implies view
    • view means present as a question that cannot be edited. Note that if a role has the 'admin' flag then this person will be able to edit this question after a confirmation.
    • use means hide from view. (e.g. may still be used in a constraint, but we don't need labels etc.)
    • If no entry for the current user's {ROLENAME} then no permission; the data is not sent to the browser.
  • dataType: string identifier of the data that is stored for this part, works alongside ask.inputType and other ask values to define the data. See reference/DataTypes.md and also PartDesigner.vue::data()

  • ask: an object that describes how the questions will be presented, at the point of asking people for answers. e.g. the question may be asked like "What’s your favourite colour?"

  • tell: an object that describes how the answers will be presented, at the point of display. e.g. the answer may be labelled with "Favourite colour"

The ask object looks like:

  • inputType: string identifier for a type of input. Valid values:
  • radios
  • checkbox
  • select
  • checkboxes
  • date
  • files
  • email
  • text
  • url
  • number
  • textarea
  • prefix, suffix: HTML or MarkDown
  • label The TEXT to be used inside a <label> element, where appropriate
  • required a constraint: this is required for submission.
  • default mixed. Optional. The default value.
  • options either a string alias "YesNo" or an array of options:
    [
      [ {STOREDVALUE}, {DISPLAYVALUE} ], ...
    ]
    

We'll need some constraints here too.

The tell object looks like:

  • prefix, suffix: HTML or MarkDown
  • label a string
  • renderer string name of a renderer function that will receive the question and answer data and have to render it. If ommitted, show as text. We will need some options here, too, e.g. if it's a link field how the link works.

Sections, Pages, Blocks

By default all questions belong to the default section on the default page, and don't belong to any blocks.

Sections, pages, blocks are identified by part entries at their start of the following data types. A new section starts a new page; a new page ends any open blocks.

Question dataType values:

  • section Start a new section
  • page Start a new page
  • groupStart Start a new group
  • groupEnd Manually ends the latest group.
  • repeatStart Start a new group of questions for which we accept sets of answers (like field collection)
  • repeatEnd Manually ends the latest repeat group.

Relevance

Parts can include a special expression whose calculated value is used to determine whether this part is relevant, based on other data in the survey. Data that is not relevant is hidden in the edit UI.

By setting relevance on a group/page/section you can hide several parts at once.

Participant

The answers field contains JSON blob keyed by question id:

{QUESTIONID}: {
  "value": {DATAVALUE},
  "saved": {LOGID},
  "locked": {LOGID},
}, ...

The {DATAVALUE} depends on the dataType in the question.

The {LOGID} points to the HarkLog entries.

The restrictions key: I suppose the idea here is for localised overrides; e.g. can we open the survey for X, or extend their deadline. @todo

Multi-choice selects / radio lists / rankings

These answer value is stored as follows:

{
  "presets": [ "key1", ... ],
  "other": { "_other_keyname": "value", ... }
}

Matrix radios/drop downs

This is where you have some common columns like "hate it" ... "love it" and a set of rows like "marmite", "marmalade"

Or the columns have a label and a set of select options.

The question would need to specify value options (that's like normal options for, say, a select input), and rowOptions, which would ideally have a codename and a display name.

The answer would be stored as { rowCodeName: optionValue, ... }

It could be queried like "find all where marmite = love it"

File uploads

File upload field can accept any number of uploaded files, limited by extension. Each upload can have an optional comment.

Files are given a unique ID which is used for its filename inside a directory at [civicrm.private]/hark/survey-{ID}/participant-{ID}/. This unique ID is considered secret for security purposes so this ID is never sent to the browser. Additionally, HTTP access to files in that directory should be disallowed - configure your web server appropriately.

File answer value is like so:

{
  "Q0123456789AB": [
    {
      "fileID": {SECRET_FILE_ID},
      "name": "Original filename.pdf",
      "mimetype": "application/pdf",
      "size": 300000,
      "comment": "Example comment",
    },
    ...
  ],
}

Check types

  • single choice drop-down.
  • single choice radio.
  • single choice checkbox. These are not good. You can't tell when it's unchecked deliberately, or just not looked at.
  • multi choice select(2)
  • multi choice checkboxes
  • Star rating: could do, basically a radio.
  • Single text box
  • Single text area
  • Matrix of drop downs
  • Matrix / rating scale
  • URLs
  • emails
  • File upload
  • Slider (meh)
  • Multiple text boxes (?no)
  • Date/Time: ?

Repeatable fields (unimplemented)

I think these are defined by the field type rather than a generic concept.

e.g. file uploads, may want to ask for 1, 3 or unlimited.

e.g. Q. do we need a user-enterable, or question-defined set of labels for the multiple ones? This sort of gets on to repeatable sections stuff. KISS

Repeatable sections (unimplemented)

The idea here (required?) would be like e.g. list your work experience, with each row having start/end date, description.

Identified by: repeatstart. Which would need something saying how many repeats you allowed.

Answer data would be stored against the id of the repeatstart and would be an array of objects, each object has the keys of the question IDs in that repeat group.

"qhash1": [
   { "qhash2": { "value": 123 }, "qhash3": { "value": 456 }, ... },
   { "qhash2": { "value": 222 }, "qhash3": { "value": 333 }, ... },
]

I don't think these are required, but I think they could be accommodated.

Sending participant/survey data to the browser

We send a filtered and pre-processed set of definitions and answers, depending on the role and possibly ux (??).

Answers are sent in the same structure except that they are filtered for access permission.

Parts are filtered and modified:

  • acl is removed. Instead, accessLevel is created with one of use|view|edit, taken from the value for the role.

  • prefix, suffix are sent as markdown and converted to HTML in the browser.

  • If the permission available to the role requesting the data is none then no data is sent.

  • If it’s use then minimal data is sent; we assume we won’t be presenting this as a part, but that its value is required by other fields.

  • If it’s view then some ask parts may be removed. (@todo check implementation here) - it might be that we include sensitive data in the edit texts that we don't want sent to the browser for a particular role. e.g. if the edit text for an assessor includes text like "the right answer is lemons. If they mention lemons, give full points." we need to make sure that does not get to them. @todo check, but I think this can be achieved by entering prefix/suffix overrides in tell

HarkParticipantContact

The participant-contributors table registers who is allowed to edit a participant's survey as a non-admin. It includes an authHash which authenticates this person via a link. This hash can be rolled at any time to invalidate a previous link, and thus deny access.

  • id (INT)
  • participant (FK INT)
  • contactID (FK INT)
  • role (STRING)
  • authHash (STRING)

Thus all such access is personal/authenticated.

As well as this sort of access, actual staff using Civi should have full access to any participant, and given a special 'admin' role via CMS permission.

Note that these records are only for providing access; they are not required for data storage. The role and contactID are copied to HarkLog entries for locks and saves, which means that somebody's role can change over time, and also means that admins can have their edits stored without needing a HarkParticipantContact entry.