model and volume server configuration and docs

docs/model-server/readme.md

@@ -6,133 +6,64 @@ Model Server is a tool for preprocessing and querying macromolecular structure d
 Installing and Running
 =====================
 
-Getting the code (use node 8+):
+Requires nodejs 8+.
+
+## From GitHub
+
 ```
 git clone https://github.com/molstar/molstar
 npm install
 ```
 
-Customize configuration at ``src/server/model/config.ts`` to point to your data and which custom properties to include (see the [Custom Properties](#custom-properties) section). Alternatively, the config can be edited in the compiled version in ``build/node_modules/servers/model/config.js``.
-
-Afterwards, build the project:
+Afterwards, build the project source:
 
 ```
-npm run build
+npm run build-tsc
 ```
 
-(or run watch mode for automatic rebuilds: ``npm run watch``)
+and run the server by 
 
-Running the server locally for testing:
 ```
-npm run model-server
-```
-or
-```
-node build/node_modules/servers/model/server
+node lib/servers/model/server/server
 ```
 
-In production it is a good idea to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever).
+## From NPM
 
+```
+npm install molstar
+./model-server 
+```
 
-## Memory issues
+(or ``node node_modules\.bin\model-server`` in Windows).
 
-Sometimes nodejs might run into problems with memory. This is usually resolved by adding the ``--max-old-space-size=8192`` parameter.
+The NPM package contains all the tools mentioned here as "binaries":
 
-Preprocessor
-============
+- ``model-server``
+- ``model-server-query``
+- ``model-server-preprocess``
 
-The preprocessor application allows to add custom data to CIF files and/or convert CIF to BinaryCIF. See the [Custom Properties](#custom-properties) section for providing custom properties.
 
-## Usage
+### Production use
 
-The app works in two modes: single files and folders.
+In production it is required to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever).
 
-Single files:
 
-```
-node build\node_modules\servers\model\preprocess -i input.cif [-oc output.cif] [-ob output.bcif] [--cfg config.json]
-```
+### Memory issues
 
-Folder:
-```
-node build\node_modules\servers\model\preprocess -fin input_folder [-foc output_cif_folder] [-fob output_bcif_folder] [--cfg config.json]
-```
+Sometimes nodejs might run into problems with memory. This is usually resolved by adding the ``--max-old-space-size=8192`` parameter.
 
-## Config
-
-The config speficies the maximum number of processes to use (in case of folder processing) and defines sources and parameters for custom properties.
-
-Example:
-```json
-{
-    "numProcesses": 4,
-    "customProperties": {
-        "sources": [
-            "./properties/pdbe"
-        ],
-        "params": {
-            "PDBe": {
-                "UseFileSource": false,
-                "API": {
-                    "residuewise_outlier_summary": "https://www.ebi.ac.uk/pdbe/api/validation/residuewise_outlier_summary/entry",
-                    "preferred_assembly": "https://www.ebi.ac.uk/pdbe/api/pdb/entry/summary",
-                    "struct_ref_domain": "https://www.ebi.ac.uk/pdbe/api/mappings/sequence_domains"
-                }
-            }
-        }
-    }
-}
-```
+## Preprocessor
 
-Custom Properties
-=================
+The preprocessor application allows to add custom data to CIF files and/or convert CIF to BinaryCIF. ``node lib/servers/model/preprocess`` or ``model-server-preprocess`` binary from the NPM package.
 
-It is possible to provide property descriptors that transform data to internal representation and define how it should be exported into one or mode CIF categories. Examples of this are located in the ``mol-model-props`` module and are linked to the server in the config and ``servers/model/properties``.
 
-Local Mode
-==========
+## Local Mode
 
-The server can be run in local/file based mode:
+The server can be run in local/file based mode using ``node lib/servers/model/query`` (``model-server-query`` binary from the NPM package).
 
-```
-node build/node_modules/servers/model/server jobs.json
-```
+Custom Properties
+=================
 
-where ``jobs.json`` is an array of
-
-```ts
-type LocalInput = {
-    input: string,
-    output: string,
-    query: QueryName,
-    modelNums?: number[],
-    params?: any,
-    binary?: boolean
-}[]
-```
+This feature is still in development.
 
-For example
-
-```json
-[
-  {
-    "input": "c:/test/quick/1tqn.cif",
-    "output": "c:/test/quick/localapi/1tqn_full.cif",
-    "query": "full"
-  },
-  {
-    "input": "c:/test/quick/1tqn.cif",
-    "output": "c:/test/quick/localapi/1tqn_full.bcif",
-    "query": "full",
-    "params": {}
-  },
-  {
-    "input": "c:/test/quick/1cbs_updated.cif",
-    "output": "c:/test/quick/localapi/1cbs_ligint.cif",
-    "query": "residueInteraction",
-    "params": {
-      "atom_site": { "label_comp_id": "REA" }
-    }
-  }
-]
-```
+It is possible to provide property descriptors that transform data to internal representation and define how it should be exported into one or mode CIF categories. Examples of this are located in the ``mol-model-props`` module and are linked to the server in the config and ``servers/model/properties``.

docs/volume-server/README.md

@@ -7,107 +7,64 @@ It uses the text based CIF and BinaryCIF formats to deliver the data to the clie
 
 For quick info about the benefits of using the server, check out the [examples](examples.md).
 
-Installing the Server 
+Installing and Running
 =====================
 
-- Install [Node.js](https://nodejs.org/en/) (tested on Node 6.* and 7.*; x64 version is strongly preferred).
-- Get the code.
-- Prepare the data.
-- Run the server.
+Requires nodejs 8+.
 
-Preparing the Data
-------------------
+## From GitHub
+
+```
+git clone https://github.com/molstar/molstar
+npm install
+```
+
+Afterwards, build the project source:
+
+```
+npm run build-tsc
+```
+
+and run the server by 
+
+```
+node lib/servers/volume/server
+```
+
+## From NPM
+
+```
+npm install molstar
+./volume-server 
+```
+
+(or ``node node_modules\.bin\volume-server`` in Windows).
+
+The NPM package contains all the tools mentioned here as "binaries":
+
+- ``volume-server``
+- ``volume-server-pack``
+- ``volume-server-query``
+
+
+### Production use
+
+In production it is required to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever).
+
+
+### Memory issues
+
+Sometimes nodejs might run into problems with memory. This is usually resolved by adding the ``--max-old-space-size=8192`` parameter.
+
+
+## Preparing the Data
 
 For the server to work, CCP4/MAP (models 0, 1, 2 are supported) input data need to be converted into a custom block format. 
-To achieve this, use the ``pack`` application.
-
-- To prepare data from x-ray based methods, use: 
-
-    ```
-    node pack -xray main.ccp4 diff.ccp4 out.mdb
-    ```
-
-- For EM data, use:
-
-    ```
-    node pack -em em.map out.mdb
-    ```
-
-Running the Server
-------------------
-
-- Install production dependencies:
-
-   ```
-   npm install --only=production
-   ```
-
-- Update ``server-config.js`` to link to your data and optionally tweak the other parameters.
-
-- Run it:
-
-    ```
-    node server
-    ```
-
-    In production it is a good idea to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever).
-
-### Local Mode
-
-The program ``local`` in the build folder can be used to query the data without running a http server.
-
-- ``node local`` prints the program usage.
-- ``node local jobs.json`` takes a list of jobs to execute in JSON format. A job entry is defined by this interface:
-
-    ```TypeScript
-    interface JobEntry {
-        source: {
-            filename: string,    
-            name: string,
-            id: string
-        },
-        query: {
-            kind: 'box' | 'cell',
-            space?: 'fractional' | 'cartesian',
-            bottomLeft?: number[],
-            topRight?: number[],
-        }
-        params: {
-            /** Determines the detail level as specified in server-config */
-            detail?: number,
-            /** 
-             * Determines the sampling level:
-             * 1: Original data
-             * 2: Downsampled by factor 1/2
-             * ...
-             * N: downsampled 1/2^(N-1)
-             */
-            forcedSamplingLevel?: number,
-            asBinary: boolean,
-        },
-        outputFolder: string
-    }
-    ```
-
-    Example ``jobs.json`` file content:
-
-    ```TypeScript
-    [{
-        source: {
-            filename: `g:/test/mdb/emd-8116.mdb`,
-            name: 'em',
-            id: '8116',
-        },
-        query: {
-            kind: 'cell'
-        },
-        params: {
-            detail: 4,
-            asBinary: true
-        },
-        outputFolder: 'g:/test/local-test'
-    }]
-    ```
+To achieve this, use the ``pack`` application (``node lib/servers/volume/pack`` or ``volume-server-pack`` binary from the NPM package).
+
+## Local Mode
+
+The program  ``lib/servers/volume/pack`` (``volume-server-query`` in NPM package) can be used to query the data without running a http server.
 
 ## Navigating the Source Code
 
@@ -122,8 +79,8 @@ The source code is split into 2 mains parts: ``pack`` and ``server``:
 Consuming the Data 
 ==================
 
-The data can be consumed in any (modern) browser using the [CIFTools.js library](https://github.com/dsehnal/CIFTools.js) (or any other piece of code that can read text or binary CIF).
+The data can be consumed in any (modern) browser using the [ciftools library](https://github.com/molstar/ciftools) (or any other piece of code that can read text or binary CIF).
 
 The [Data Format](DataFormat.md) document gives a detailed description of the server response format.
 
-As a reference/example of the server usage, please see the implementation in [LiteMol](https://github.com/dsehnal/LiteMol) ([CIF.ts + Data.ts](https://github.com/dsehnal/LiteMol/tree/master/src/lib/Core/Formats/Density), [UI](https://github.com/dsehnal/LiteMol/tree/master/src/Viewer/Extensions/DensityStreaming)) or in Mol*.
+As a reference/example of the server usage is available in Mol* ``mol-plugin`` module.

package.json

@@ -26,7 +26,7 @@
     "serve": "http-server -p 1338",
     "model-server": "node lib/servers/model/server.js",
     "model-server-watch": "nodemon --watch lib lib/servers/model/server.js",
-    "volume-server": "node lib/servers/volume/server.js --idMap em 'test/${id}.mdb' --defaultPort 1336",
+    "volume-server-test": "node lib/servers/volume/server.js --idMap em 'test/${id}.mdb' --defaultPort 1336",
     "plugin-state": "node lib/servers/plugin-state/index.js",
     "preversion": "npm run test",
     "postversion": "git push && git push --tags",
@@ -35,6 +35,14 @@
   "files": [
     "lib/"
   ],
+  "bin": {
+    "model-server": "lib/servers/model/server.js",
+    "model-server-query": "lib/servers/model/local.js",
+    "model-server-preprocess": "lib/servers/model/preprocess.js",
+    "volume-server": "lib/servers/volume/server.js",
+    "volume-server-query": "lib/servers/volume/query.js",
+    "volume-server-pack": "lib/servers/volume/pack.js"
+  },
   "nodemonConfig": {
     "ignoreRoot": [
       "./node_modules",

src/servers/model/config.ts

@@ -1,18 +1,19 @@
 /**
- * Copyright (c) 2018-2019 mol* contributors, licensed under MIT, See LICENSE file for more info.
+ * Copyright (c) 2018-2020 mol* contributors, licensed under MIT, See LICENSE file for more info.
  *
  * @author David Sehnal <david.sehnal@gmail.com>
  */
 
+import * as argparse from 'argparse'
+import { ObjectKeys } from '../../mol-util/type-helpers'
+import VERSION from './version'
+import * as fs from 'fs'
+import { ModelPropertyProviderConfig } from './property-provider';
+
 const DefaultModelServerConfig = {
-    /**
-     * Determine if and how long to cache entries after a request.
-     */
-    cacheParams: {
-        useCache: true,
-        maxApproximateSizeInBytes: 2 * 1014 * 1024 * 1024, // 2 GB
-        entryTimeoutInMs: 10 * 60 * 1000 // 10 minutes
-    },
+    // 0 for off
+    cacheMaxSizeInBytes: 2 * 1014 * 1024 * 1024, // 2 GB
+    cacheEntryTimeoutMs: 10 * 60 * 1000, // 10 minutes
 
     /**
      * Node (V8) sometimes exhibits GC related issues  that significantly slow down the execution
@@ -22,13 +23,11 @@ const DefaultModelServerConfig = {
      * For this to work, the server must be run using a deamon (i.e. forever.js on Linux
      * or IISnode on Windows) so that the server is automatically restarted when the shutdown happens.
      */
-    shutdownParams: {
-        // 0 for off, server will shut down after this amount of minutes.
-        timeoutMinutes: 24 * 60 /* a day */,
 
-        // modifies the shutdown timer by +/- timeoutVarianceMinutes (to avoid multiple instances shutting at the same time)
-        timeoutVarianceMinutes: 60
-    },
+    // 0 for off, server will shut down after this amount of minutes.
+    shutdownTimeoutMinutes: 24 * 60, /* a day */
+    // modifies the shutdown timer by +/- timeoutVarianceMinutes (to avoid multiple instances shutting at the same time)
+    shutdownTimeoutVarianceMinutes: 60,
 
     defaultPort: 1337,
 
@@ -36,13 +35,13 @@ const DefaultModelServerConfig = {
      * Specify the prefix of the API, i.e.
      * <host>/<apiPrefix>/<API queries>
      */
-    appPrefix: '/ModelServer',
+    apiPrefix: '/ModelServer',
 
     /**
      * The maximum time the server dedicates to executing a query.
      * Does not include the time it takes to read and export the data.
      */
-    maxQueryTimeInMs: 5 * 1000,
+    queryTimeoutMs: 5 * 1000,
 
     /** Maximum number of requests before "server busy" */
     maxQueueLength: 30,
@@ -51,7 +50,7 @@ const DefaultModelServerConfig = {
      * Provide a property config or a path a JSON file with the config.
      */
     // TODO: finish customProperty support
-    customProperties: <import('./property-provider').ModelPropertyProviderConfig | string>{
+    customProperties: <ModelPropertyProviderConfig | string>{
         sources: [
             // 'pdbe',
             // 'rcsb',
@@ -91,26 +90,164 @@ const DefaultModelServerConfig = {
      *
      * /static query uses 'pdb-cif' and 'pdb-bcif' source names.
      */
-    fileMapping: [
+    sourceMap: [
         ['pdb-cif', 'e:/test/quick/${id}_updated.cif'],
         // ['pdb-bcif', 'e:/test/quick/${id}.bcif'],
     ] as [string, string][]
 };
 
+export let mapSourceAndIdToFilename: (source: string, id: string) => string = () => {
+    throw new Error('call setupConfig & validateConfigAndSetupSourceMap to initialize this function');
+}
+
+function addServerArgs(parser: argparse.ArgumentParser) {
+    parser.addArgument([ '--apiPrefix' ], {
+        defaultValue: DefaultModelServerConfig.apiPrefix,
+        metavar: 'PREFIX',
+        help: `Specify the prefix of the API, i.e. <host>/<apiPrefix>/<API queries>`
+    });
+    parser.addArgument([ '--defaultPort' ], {
+        defaultValue: DefaultModelServerConfig.defaultPort,
+        metavar: 'PORT',
+        type: 'int',
+        help: `Specify the port the server is running on`
+    });
+    parser.addArgument([ '--cacheMaxSizeInBytes' ], {
+        defaultValue: DefaultModelServerConfig.cacheMaxSizeInBytes,
+        metavar: 'CACHE_SIZE',
+        type: 'int',
+        help: `0 for off.`
+    });
+    parser.addArgument([ '--cacheEntryTimeoutMs' ], {
+        defaultValue: DefaultModelServerConfig.cacheEntryTimeoutMs,
+        metavar: 'CACHE_TIMEOUT',
+        type: 'int',
+        help: `Specify in ms how long to keep entries in cache.`
+    });
+    parser.addArgument([ '--queryTimeoutMs' ], {
+        defaultValue: DefaultModelServerConfig.queryTimeoutMs,
+        metavar: 'QUERY_TIMEOUT',
+        type: 'int',
+        help: `The maximum time the server dedicates to executing a query in ms.\nDoes not include the time it takes to read and export the data.`
+    });
+    parser.addArgument([ '--shutdownTimeoutMinutes' ], {
+        defaultValue: DefaultModelServerConfig.shutdownTimeoutMinutes,
+        metavar: 'TIME',
+        type: 'int',
+        help: `0 for off, server will shut down after this amount of minutes.`
+    });
+    parser.addArgument([ '--shutdownTimeoutVarianceMinutes' ], {
+        defaultValue: DefaultModelServerConfig.shutdownTimeoutVarianceMinutes,
+        metavar: 'VARIANCE',
+        type: 'int',
+        help: `modifies the shutdown timer by +/- timeoutVarianceMinutes (to avoid multiple instances shutting at the same time)`
+    });
+    parser.addArgument([ '--defaultSource' ], {
+        defaultValue: DefaultModelServerConfig.defaultSource,
+        metavar: 'DEFAULT_SOURCE',
+        help: `modifies which 'sourceMap' source to use by default` 
+    });
+    parser.addArgument([ '--sourceMap' ], {
+        nargs: 2,
+        action: 'append',
+        metavar: ['SOURCE', 'PATH'] as any,
+        help: [
+            'Map `id`s for a `source` to a file path.',
+            'Example: pdb-bcif \'../../data/bcif/${id}.bcif\' ',
+            'JS expressions can be used inside ${}, e.g. \'${id.substr(1, 2)}/${id}.mdb\'',
+            'Can be specified multiple times.',
+            'The `SOURCE` variable (e.g. `pdb-bcif`) is arbitrary and depends on how you plan to use the server.'
+        ].join('\n'),
+    });
+}
+
 export type ModelServerConfig = typeof DefaultModelServerConfig
-export const ModelServerConfig = DefaultModelServerConfig
+export const ModelServerConfig = { ...DefaultModelServerConfig }
 
-export let mapSourceAndIdToFilename: (source: string, id: string) => string = () => {
-    throw new Error('call setupConfig to initialize this function');
+export const ModelServerConfigTemplate: ModelServerConfig = { 
+    ...DefaultModelServerConfig,
+    defaultSource: 'pdb-bcif',
+    sourceMap: [
+        ['pdb-bcif', './path-to-binary-cif/${id.substr(1, 2)}/${id}.bcif'],
+        ['pdb-cif', './path-to-text-cif/${id.substr(1, 2)}/${id}.cif'],
+        ['pdb-updated', './path-to-updated-cif/${id}.bcif']
+    ] as [string, string][]
 }
 
-export function setupConfig(cfg?: ModelServerConfig) {
-    if (cfg) Object.assign(ModelServerConfig, cfg);
-    if (!ModelServerConfig.fileMapping) return;
+// TODO: include once properly supported
+delete ModelServerConfigTemplate.customProperties
 
+interface ServerJsonConfig {
+    cfg?: string,
+    printCfg?: any,
+    cfgTemplate?: any
+}
+
+function addJsonConfigArgs(parser: argparse.ArgumentParser) {
+    parser.addArgument(['--cfg'], { 
+        help: [
+            'JSON config file path',
+            'If a property is not specified, cmd line param/OS variable/default value are used.'
+        ].join('\n'),
+        required: false 
+    });
+    parser.addArgument(['--printCfg'], { help: 'Print current config for validation and exit.', required: false, nargs: 0 });
+    parser.addArgument(['--cfgTemplate'], { help: 'Prints default JSON config template to be modified and exits.', required: false, nargs: 0 });
+}
+
+function setConfig(config: ModelServerConfig) {
+    for (const k of ObjectKeys(ModelServerConfig)) {
+        if (config[k] !== void 0) (ModelServerConfig as any)[k] = config[k];
+    }
+}
+
+function validateConfigAndSetupSourceMap() {
+    if (!ModelServerConfig.sourceMap || ModelServerConfig.sourceMap.length === 0) {
+        throw new Error(`Please provide 'sourceMap' configuration. See [-h] for available options.`);
+    }
+    
     mapSourceAndIdToFilename = new Function('source', 'id', [
         'switch (source.toLowerCase()) {',
-        ...ModelServerConfig.fileMapping.map(([source, path]) => `case '${source.toLowerCase()}': return \`${path}\`;`),
+        ...ModelServerConfig.sourceMap.map(([source, path]) => `case '${source.toLowerCase()}': return \`${path}\`;`),
         '}',
     ].join('\n')) as any;
+}
+
+function parseConfigArguments() {
+    const parser = new argparse.ArgumentParser({
+        version: VERSION,
+        addHelp: true,
+        description: `ModelServer ${VERSION}, (c) 2018-2020, Mol* contributors`
+    });
+    addJsonConfigArgs(parser);
+    addServerArgs(parser);
+    return parser.parseArgs() as ModelServerConfig & ServerJsonConfig;
+}
+
+export function configureServer() {
+    const config = parseConfigArguments();
+
+    if (config.cfgTemplate !== null) {
+        console.log(JSON.stringify(ModelServerConfigTemplate, null, 2));
+        process.exit(0);
+    }
+    
+    try {
+        setConfig(config) // sets the config for global use
+    
+        if (config.cfg) {
+            const cfg = JSON.parse(fs.readFileSync(config.cfg, 'utf8')) as ModelServerConfig;
+            setConfig(cfg);
+        }
+    
+        if (config.printCfg !== null) {
+            console.log(JSON.stringify(ModelServerConfig, null, 2));
+            process.exit(0);
+        }
+    
+        validateConfigAndSetupSourceMap();
+    } catch (e) {
+        console.error('' + e);
+        process.exit(1);
+    }    
 }

src/servers/model/local.ts → src/servers/model/query.ts

@@ -8,7 +8,7 @@ import * as fs from 'fs'
 import Version from './version';
 import { LocalInput, runLocal } from './server/api-local';
 
-console.log(`Mol* ModelServer (${Version}), (c) 2018 Mol* authors`);
+console.log(`Mol* ModelServer (${Version}), (c) 2018-2020 Mol* authors`);
 console.log(``);
 
 let exampleWorkload: LocalInput = [{

src/servers/model/server.ts

@@ -4,25 +4,23 @@
  * @author David Sehnal <david.sehnal@gmail.com>
  */
 
-import * as express from 'express'
 import * as compression from 'compression'
-import * as fs from 'fs'
-import * as argparse from 'argparse'
-import { ModelServerConfig as ServerConfig, setupConfig, ModelServerConfig } from './config'
-import { ConsoleLogger } from '../../mol-util/console-logger';
-import { PerformanceMonitor } from '../../mol-util/performance-monitor';
-import { initWebApi } from './server/api-web';
+import * as express from 'express'
+import { ConsoleLogger } from '../../mol-util/console-logger'
+import { PerformanceMonitor } from '../../mol-util/performance-monitor'
+import { configureServer, ModelServerConfig as ServerConfig } from './config'
+import { initWebApi } from './server/api-web'
 import Version from './version'
 
 function setupShutdown() {
-    if (ServerConfig.shutdownParams.timeoutVarianceMinutes > ServerConfig.shutdownParams.timeoutMinutes) {
+    if (ServerConfig.shutdownTimeoutVarianceMinutes > ServerConfig.shutdownTimeoutMinutes) {
         ConsoleLogger.log('Server', 'Shutdown timeout variance is greater than the timer itself, ignoring.');
     } else {
         let tVar = 0;
-        if (ServerConfig.shutdownParams.timeoutVarianceMinutes > 0) {
-            tVar = 2 * (Math.random() - 0.5) * ServerConfig.shutdownParams.timeoutVarianceMinutes;
+        if (ServerConfig.shutdownTimeoutVarianceMinutes > 0) {
+            tVar = 2 * (Math.random() - 0.5) * ServerConfig.shutdownTimeoutVarianceMinutes;
         }
-        let tMs = (ServerConfig.shutdownParams.timeoutMinutes + tVar) * 60 * 1000;
+        let tMs = (ServerConfig.shutdownTimeoutMinutes + tVar) * 60 * 1000;
 
         console.log(`----------------------------------------------------------------------------`);
         console.log(`  The server will shut down in ${PerformanceMonitor.format(tMs)} to prevent slow performance.`);
@@ -41,22 +39,7 @@ function setupShutdown() {
     }
 }
 
-const cmdParser = new argparse.ArgumentParser({
-    addHelp: true
-});
-
-cmdParser.addArgument(['--cfg'], { help: 'Config file path.', required: false });
-cmdParser.addArgument(['--printcfg'], { help: 'Prints out current config and exits.', required: false, nargs: 0 });
-
-interface CmdArgs {
-    cfg?: string,
-    printcfg?: boolean
-}
-
-const cmdArgs = cmdParser.parseArgs() as CmdArgs;
-
-const cfg = cmdArgs.cfg ? JSON.parse(fs.readFileSync(cmdArgs.cfg, 'utf8')) : void 0;
-setupConfig(cfg);
+configureServer();
 
 function startServer() {
     let app = express();
@@ -73,12 +56,8 @@ function startServer() {
     console.log(``);
 }
 
-if (cmdArgs.printcfg !== null) {
-    console.log(JSON.stringify(ModelServerConfig, null, 2));
-} else {
-    startServer();
+startServer();
 
-    if (ServerConfig.shutdownParams && ServerConfig.shutdownParams.timeoutMinutes > 0) {
-        setupShutdown();
-    }
+if (ServerConfig.shutdownTimeoutMinutes > 0) {
+    setupShutdown();
 }

src/servers/model/server/api-schema.ts

@@ -44,7 +44,7 @@ export function getApiSchema() {
 function getPaths() {
     const ret: any = {};
     for (const { name, definition } of QueryList) {
-        ret[`${ServerConfig.appPrefix}/v1/{id}/${name}`] = getQueryInfo(definition);
+        ret[`${ServerConfig.apiPrefix}/v1/{id}/${name}`] = getQueryInfo(definition);
     }
     return ret;
 }

src/servers/model/server/api-web.ts

@@ -18,7 +18,7 @@ import { getApiSchema, shortcutIconLink } from './api-schema';
 import { swaggerUiAssetsHandler, swaggerUiIndexHandler } from '../../common/swagger-ui';
 
 function makePath(p: string) {
-    return Config.appPrefix + '/' + p;
+    return Config.apiPrefix + '/' + p;
 }
 
 function wrapResponse(fn: string, res: express.Response) {
@@ -187,7 +187,7 @@ export function initWebApi(app: express.Express) {
     app.use(makePath(''), swaggerUiAssetsHandler());
     app.get(makePath(''), swaggerUiIndexHandler({
         openapiJsonUrl: makePath('openapi.json'),
-        apiPrefix: Config.appPrefix,
+        apiPrefix: Config.apiPrefix,
         title: 'ModelServer API',
         shortcutIconLink
     }));

src/servers/model/server/cache.ts

@@ -48,7 +48,7 @@ export class Cache<T> {
     private refresh(e: CacheNode<T>) {
         this.clearTimeout(e);
 
-        e.value.timeoutId = setTimeout(() => this.expireNode(e), ServerConfig.cacheParams.entryTimeoutInMs);
+        e.value.timeoutId = setTimeout(() => this.expireNode(e), ServerConfig.cacheEntryTimeoutMs);
         this.entries.remove(e);
         this.entries.addFirst(e.value);
     }
@@ -74,7 +74,7 @@ export class Cache<T> {
 
         if (this.entryMap.has(key)) this.dispose(this.entryMap.get(key)!);
 
-        if (ServerConfig.cacheParams.maxApproximateSizeInBytes < this.approximateSize + approximateSize) {
+        if (ServerConfig.cacheMaxSizeInBytes < this.approximateSize + approximateSize) {
             if (this.entries.last) this.dispose(this.entries.last);
         }
 

src/servers/model/server/query.ts

@@ -57,7 +57,7 @@ export async function resolveJob(job: Job): Promise<CifWriter.Encoder<any>> {
         const queries = structures.map(s => job.queryDefinition.query(job.normalizedParams, s));
         const result: Structure[] = [];
         for (let i = 0; i < structures.length; i++) {
-            const s = await StructureSelection.unionStructure(StructureQuery.run(queries[i], structures[i], { timeoutMs: Config.maxQueryTimeInMs }))
+            const s = await StructureSelection.unionStructure(StructureQuery.run(queries[i], structures[i], { timeoutMs: Config.queryTimeoutMs }))
             if (s.elementCount > 0) result.push(s);
         }
         perf.end('query');
@@ -112,7 +112,7 @@ function doError(job: Job, e: any) {
     return encoder;
 }
 
-const maxTime = Config.maxQueryTimeInMs;
+const maxTime = Config.queryTimeoutMs;
 export function abortingObserver(p: Progress) {
     if (now() - p.root.progress.startedTime > maxTime) {
         p.requestAbort(`Exceeded maximum allowed time for a query (${maxTime}ms)`);

src/servers/model/server/structure-wrapper.ts

@@ -49,12 +49,12 @@ export interface StructureWrapper {
 }
 
 export async function createStructureWrapperFromJob(job: Job, propertyProvider: ModelPropertiesProvider | undefined, allowCache = true): Promise<StructureWrapper> {
-    if (allowCache && Config.cacheParams.useCache) {
+    if (allowCache && Config.cacheMaxSizeInBytes > 0) {
         const ret = StructureCache.get(job.key);
         if (ret) return ret;
     }
     const ret = await readStructureWrapper(job.key, job.sourceId, job.entryId, propertyProvider);
-    if (allowCache && Config.cacheParams.useCache) {
+    if (allowCache && Config.cacheMaxSizeInBytes > 0) {
         StructureCache.add(ret);
     }
     return ret;

src/servers/volume/config.ts

@@ -1,14 +1,24 @@
 /**
- * Copyright (c) 2019 mol* contributors, licensed under MIT, See LICENSE file for more info.
+ * Copyright (c) 2019-2020 mol* contributors, licensed under MIT, See LICENSE file for more info.
  *
  * @author David Sehnal <david.sehnal@gmail.com>
  * @author Alexander Rose <alexander.rose@weirdbyte.de>
  */
 
 import * as argparse from 'argparse'
-import { ObjectKeys } from '../../mol-util/type-helpers';
+import { ObjectKeys } from '../../mol-util/type-helpers'
+import { VOLUME_SERVER_HEADER, VOLUME_SERVER_VERSION } from './server/version'
+import * as fs from 'fs'
 
-export function addLimitsArgs(parser: argparse.ArgumentParser) {
+const DefaultServerConfig = {
+    apiPrefix: '/VolumeServer',
+    defaultPort: 1337,
+    shutdownTimeoutMinutes: 24 * 60, /* a day */
+    shutdownTimeoutVarianceMinutes: 60,
+    idMap: [] as [string, string][]
+}
+
+function addLimitsArgs(parser: argparse.ArgumentParser) {
     parser.addArgument([ '--maxRequestBlockCount' ], {
         defaultValue: DefaultLimitsConfig.maxRequestBlockCount,
         metavar: 'COUNT',
@@ -32,7 +42,7 @@ The maximum number of voxels is tied to maxRequestBlockCount.`
     });
 }
 
-export function addServerArgs(parser: argparse.ArgumentParser) {
+function addServerArgs(parser: argparse.ArgumentParser) {
     parser.addArgument([ '--apiPrefix' ], {
         defaultValue: DefaultServerConfig.apiPrefix,
         metavar: 'PREFIX',
@@ -65,15 +75,15 @@ export function addServerArgs(parser: argparse.ArgumentParser) {
             'Map `id`s for a `type` to a file path.',
             'Example: x-ray \'../../data/mdb/xray/${id}-ccp4.mdb\'',
             '',
-            '  - JS expressions can be used with in the ${}, e.g. \'${id.substr(1, 2)}/${id}.mdb\'',
+            '  - JS expressions can be used inside ${}, e.g. \'${id.substr(1, 2)}/${id}.mdb\'',
             '  - Can be specified multiple times.',
-            '  - The "TYPE" variable (e.g. "x-ray") is arbitrary and depends on how you plan to use the server.',
-            '    By default, Mol* Viewer uses "x-ray" and "em", but any particular use case may vary. '
+            '  - The `TYPE` variable (e.g. `x-ray`) is arbitrary and depends on how you plan to use the server.',
+            '    By default, Mol* Viewer uses `x-ray` and `em`, but any particular use case may vary. '
         ].join('\n'),
     });
 }
 
-export function addJsonConfigArgs(parser: argparse.ArgumentParser) {
+function addJsonConfigArgs(parser: argparse.ArgumentParser) {
     parser.addArgument(['--cfg'], { 
         help: [
             'JSON config file path',
@@ -82,32 +92,25 @@ export function addJsonConfigArgs(parser: argparse.ArgumentParser) {
         required: false 
     });
     parser.addArgument(['--printCfg'], { help: 'Print current config for validation and exit.', required: false, nargs: 0 });
-    parser.addArgument(['--printCfgTemplate'], { help: 'Prints default JSON config template to be modified and exits.', required: false, nargs: 0 });
+    parser.addArgument(['--cfgTemplate'], { help: 'Prints default JSON config template to be modified and exits.', required: false, nargs: 0 });
 }
 
 export interface ServerJsonConfig {
     cfg?: string,
     printCfg?: any,
-    printCfgTemplate?: any
+    cfgTemplate?: any
 }
 
-const DefaultServerConfig = {
-    apiPrefix: '/VolumeServer',
-    defaultPort: 1337,
-    shutdownTimeoutMinutes: 24 * 60, /* a day */
-    shutdownTimeoutVarianceMinutes: 60,
-    idMap: [] as [string, string][]
-}
 export type ServerConfig = typeof DefaultServerConfig
 export const ServerConfig = { ...DefaultServerConfig }
 
-export function setServerConfig(config: ServerConfig) {
+function setServerConfig(config: ServerConfig) {
     for (const k of ObjectKeys(ServerConfig)) {
-        if (config[k]) (ServerConfig as any)[k] = config[k];
+        if (config[k] !== void 0) (ServerConfig as any)[k] = config[k];
     }
 }
 
-export function validateServerConfig() {
+function validateServerConfig() {
     if (!ServerConfig.idMap || ServerConfig.idMap.length === 0) {
         throw new Error(`Please provide 'idMap' configuration. See [-h] for available options.`);
     }
@@ -128,18 +131,21 @@ const DefaultLimitsConfig = {
 }
 export type LimitsConfig = typeof DefaultLimitsConfig
 export const LimitsConfig = { ...DefaultLimitsConfig }
-export function setLimitsConfig(config: LimitsConfig) {
+
+function setLimitsConfig(config: LimitsConfig) {
     for (const k of ObjectKeys(LimitsConfig)) {
-        if (config[k]) (LimitsConfig as any)[k] = config[k];
+        if (config[k] !== void 0) (LimitsConfig as any)[k] = config[k];
     }
 }
 
-export function setConfig(config: ServerConfig & LimitsConfig) {
+type FullServerConfig = ServerConfig & LimitsConfig
+
+function setConfig(config: FullServerConfig) {
     setServerConfig(config)
     setLimitsConfig(config)
 }
 
-export const ServerConfigTemplate: ServerConfig & LimitsConfig = {
+const ServerConfigTemplate: FullServerConfig = {
     ...DefaultServerConfig,
     idMap: [
         ['x-ray', './path-to-xray-data/${id.substr(1, 2)}/${id}.mdb'],
@@ -147,3 +153,77 @@ export const ServerConfigTemplate: ServerConfig & LimitsConfig = {
     ] as [string, string][],
     ...DefaultLimitsConfig
 }
+
+export function configureServer() {
+    const parser = new argparse.ArgumentParser({
+        version: VOLUME_SERVER_VERSION,
+        addHelp: true,
+        description: VOLUME_SERVER_HEADER
+    });
+    addJsonConfigArgs(parser);
+    addServerArgs(parser);
+    addLimitsArgs(parser);
+    const config = parser.parseArgs() as FullServerConfig & ServerJsonConfig;
+
+    if (config.cfgTemplate !== null) {
+        console.log(JSON.stringify(ServerConfigTemplate, null, 2));
+        process.exit(0);
+    }
+    
+    try {
+        setConfig(config) // sets the config for global use
+    
+        if (config.cfg) {
+            const cfg = JSON.parse(fs.readFileSync(config.cfg, 'utf8')) as FullServerConfig;
+            setConfig(cfg);
+        }
+    
+        if (config.printCfg !== null) {
+            console.log(JSON.stringify({ ...ServerConfig, ...LimitsConfig }, null, 2));
+            process.exit(0);
+        }
+    
+        validateServerConfig();
+    } catch (e) {
+        console.error('' + e);
+        process.exit(1);
+    }    
+}
+
+export function configureLocal() {
+    const parser = new argparse.ArgumentParser({
+        version: VOLUME_SERVER_VERSION,
+        addHelp: true,
+        description: VOLUME_SERVER_HEADER
+    });
+    parser.addArgument(['--jobs'], { help: `Path to a JSON file with job specification.`, required: false });
+    parser.addArgument(['--jobsTemplate'], { help: 'Print example template for jobs.json and exit.', required: false, nargs: 0 });;
+    addJsonConfigArgs(parser);
+    addLimitsArgs(parser);
+
+    const config = parser.parseArgs() as LimitsConfig & ServerJsonConfig;
+
+    if (config.cfgTemplate !== null) {
+        console.log(JSON.stringify(DefaultLimitsConfig, null, 2));
+        process.exit(0);
+    }
+    
+    try {
+        setLimitsConfig(config) // sets the config for global use
+    
+        if (config.cfg) {
+            const cfg = JSON.parse(fs.readFileSync(config.cfg, 'utf8')) as FullServerConfig;
+            setLimitsConfig(cfg);
+        }
+    
+        if (config.printCfg !== null) {
+            console.log(JSON.stringify(LimitsConfig, null, 2));
+            process.exit(0);
+        }
+    
+        return config as LimitsConfig & { jobs: string, jobsTemplate: any };
+    } catch (e) {
+        console.error('' + e);
+        process.exit(1);
+    }    
+}

src/servers/volume/local.ts → src/servers/volume/query.ts

@@ -7,16 +7,13 @@
  * @author Alexander Rose <alexander.rose@weirdbyte.de>
  */
 
-import * as argparse from 'argparse'
-import * as LocalApi from './server/local-api'
-import VERSION from './server/version'
-import * as fs from 'fs'
-import { LimitsConfig, addLimitsArgs, setLimitsConfig } from './config';
+import * as fs from 'fs';
+import { configureLocal } from './config';
+import * as LocalApi from './server/local-api';
 
-console.log(`VolumeServer Local ${VERSION}, (c) 2018-2019, Mol* contributors`);
-console.log();
+const config = configureLocal();
 
-function description() {
+if (config.jobsTemplate !== null) {
     const exampleJobs: LocalApi.JobEntry[] = [{
         source: {
             filename: `g:/test/mdb/xray-1tqn.mdb`,
@@ -49,29 +46,20 @@ function description() {
         },
         outputFolder: 'g:/test/local-test'
     }];
-
-    return `Usage: node local jobs.json\n\nExample jobs.json: ${JSON.stringify(exampleJobs, null, 2)}`
+    console.log(JSON.stringify(exampleJobs, null, 2));
+    process.exit();
 }
 
-const parser = new argparse.ArgumentParser({
-    addHelp: true,
-    description: description()
-});
-addLimitsArgs(parser)
-parser.addArgument(['jobs'], {
-    help: `Path to jobs JSON file.`
-})
-
-const config: LimitsConfig & { jobs: string } = parser.parseArgs()
-setLimitsConfig(config) // sets the config for global use
-
 async function run() {
     let jobs: LocalApi.JobEntry[];
     try {
+        if (!config.jobs) {
+            throw new Error(`Please provide 'jobs' argument. See [-h] for help.`);
+        }
+
         jobs = JSON.parse(fs.readFileSync(config.jobs, 'utf-8'));
     } catch (e) {
-        console.log('Error:');
-        console.error(e);
+        console.error('' + e);
         return;
     }
 

src/servers/volume/server.ts

@@ -7,16 +7,14 @@
  * @author Alexander Rose <alexander.rose@weirdbyte.de>
  */
 
-import * as express from 'express'
 import * as compression from 'compression'
-
-import init from './server/web-api'
-import VERSION from './server/version'
+import * as express from 'express'
 import { ConsoleLogger } from '../../mol-util/console-logger'
+import { configureServer, ServerConfig } from './config'
 import { State } from './server/state'
-import { addServerArgs, addLimitsArgs, LimitsConfig, setConfig, ServerConfig, addJsonConfigArgs, ServerJsonConfig, ServerConfigTemplate, validateServerConfig } from './config';
-import * as argparse from 'argparse'
-import * as fs from 'fs'
+import { VOLUME_SERVER_HEADER } from './server/version'
+import init from './server/web-api'
+
 
 function setupShutdown() {
     if (ServerConfig.shutdownTimeoutVarianceMinutes > ServerConfig.shutdownTimeoutMinutes) {
@@ -45,39 +43,7 @@ function setupShutdown() {
     }
 }
 
-const parser = new argparse.ArgumentParser({
-    addHelp: true,
-    description: `VolumeServer ${VERSION}, (c) 2018-2019, Mol* contributors`
-});
-addJsonConfigArgs(parser);
-addServerArgs(parser)
-addLimitsArgs(parser)
-
-const config: ServerConfig & LimitsConfig & ServerJsonConfig = parser.parseArgs()
-
-if (config.printCfgTemplate !== null) {
-    console.log(JSON.stringify(ServerConfigTemplate, null, 2));
-    process.exit(0);
-}
-
-try {
-    setConfig(config) // sets the config for global use
-
-    if (config.cfg) {
-        const cfg = JSON.parse(fs.readFileSync(config.cfg, 'utf8')) as ServerConfig & LimitsConfig;
-        setConfig(cfg);
-    }
-
-    if (config.printCfg !== null) {
-        console.log(JSON.stringify({ ...ServerConfig, ...LimitsConfig }, null, 2));
-        process.exit(0);
-    }
-
-    validateServerConfig();
-} catch (e) {
-    console.error('' + e);
-    process.exit(1);
-}
+configureServer();
 
 const port = process.env.port || ServerConfig.defaultPort;
 
@@ -87,11 +53,11 @@ init(app);
 
 app.listen(port);
 
-console.log(`VolumeServer ${VERSION}, (c) 2018-2019, Mol* contributors`);
+console.log(VOLUME_SERVER_HEADER);
 console.log(``);
 console.log(`The server is running on port ${port}.`);
 console.log(``);
 
-if (config.shutdownTimeoutMinutes > 0) {
+if (ServerConfig.shutdownTimeoutMinutes > 0) {
     setupShutdown();
 }

src/servers/volume/server/query/encode.ts

@@ -9,7 +9,7 @@
 import { CifWriter } from '../../../../mol-io/writer/cif'
 import * as Data from './data-model'
 import * as Coords from '../algebra/coordinate'
-import VERSION from '../version'
+import { VOLUME_SERVER_VERSION as VERSION } from '../version'
 import * as DataFormat from '../../common/data-format'
 import { Column } from '../../../../mol-data/db';
 import { ArrayEncoding, ArrayEncoder } from '../../../../mol-io/common/binary-cif';

src/servers/volume/server/version.ts

@@ -1 +1,2 @@
-export default '0.9.5'
+export const VOLUME_SERVER_VERSION = '0.9.5'
+export const VOLUME_SERVER_HEADER = `VolumeServer ${VOLUME_SERVER_VERSION}, (c) 2018-2020, Mol* contributors`

src/servers/volume/server/web-schema.ts

@@ -5,7 +5,7 @@
  * @author David Sehnal <david.sehnal@gmail.com>
  */
 
-import VERSION from './version'
+import { VOLUME_SERVER_VERSION } from './version'
 import { LimitsConfig, ServerConfig } from '../config';
 
 export function getSchema() {
@@ -18,7 +18,7 @@ export function getSchema() {
     return {
         openapi: '3.0.0',
         info: {
-            version: VERSION,
+            version: VOLUME_SERVER_VERSION,
             title: 'Volume Server',
             description: 'The VolumeServer is a service for accessing subsets of volumetric data. It automatically downsamples the data depending on the volume of the requested region to reduce the bandwidth requirements and provide near-instant access to even the largest data sets.',
         },