diff options
72 files changed, 229 insertions, 7790 deletions
@@ -1,41 +1,9 @@ -NEWS for Qvarn (JSONB) +NEWS for Salami ============================================================================= -This file has release notes for Qvarn (the JSONB version). +This file has release notes for Salami -Version 0.86+git, not yet released +Version 0.0+git, not yet released ---------------------------------- -Bug fixes: - -* Listeners for all resource types were listed for all resource types. - Thus, `GET /foos/listners` would list listeners also for `bar` - resources, not just `foo`. This has been fixed. Bug reported by - Kaius, fixed by Lars. - -* Mantas reported that if a resource type or resource had two fields - with the same type but values that were not possible to compare, - such as a string and an integer, Qvarn would crash. This is now - fixed. - -* Saulius Žemaitaitis reported that updating a resource type - specification YAML file and restarting Qvarn wouldn't make the new - version of the resource type availables. Fixed now. - -Version 0.86, released 2017-10-10 ----------------------------------- - -* All old API tests (`test-api`) pass. Qvarn (JSONB) is functionally - complete now, and API compatible with the old Qvarn. It may not be - performant enough yet. - -Version 0.84, released 2017-09-27 ------------------------------------------------------------------------------ - -* This is the first release of Qvarn (JSONB), and has been rewritten - from scratch to use the Postgres JSONB data type for storing - resources. It is NOT yet ready for production use. This release is - done so we can test Ansible modifications, run benchmarks, and other - such things. - - Do not use this release. +First release. @@ -1,38 +1,13 @@ -README for qvarn-jsonb +README for Salami ============================================================================= -This is the JSONB-using version of [Qvarn][], a web application to -manage structured data via a RESTful HTTP JSON API. It is primarily -meant for personal data, and is meant to make it easier to comply with -the EU [General Data Protection Regulation][]. - -[Qvarn]: http://qvarn.org/ -[General Data Protection Regulation]: https://en.wikipedia.org/wiki/General_Data_Protection_Regulation - -History and background ------------------------------------------------------------------------------ - -Qvarn started as an internal project in 2015 at Suomen Tilaajavastuu -(https://www.tilaajavastuu.fi/en/), a company that provides various -reproting services to the Finnish construction industry. Later that -year the Swedish Construction Federation -(https://www.sverigesbyggindustrier.se/english) adopted Qvarn as a -platform for ID06 identity card services in Sweden. From the start, a -goal for Qvarn has been to make it easy to comply with the EU General -Data Protection Regulation. - -In 2016 Qvarn spun off to its own company, QvarnLabs Ab, which -provides consulting, development, and support services around Qvarn. - Legalese ----------------------------------------------------------------------------- Qvarn in its entirety is copyright by its authorss, and released under the GNU Affero General Public Licence, version 3, or later. - Qvarn-JSONB, a RESTful HTTP API for storing structured personal information - Copyright (C) 2015, 2016 Suomen Tilaajavastuu Oy - Copyright (C) 2017 QvarnLabs Ab + Salami Copyright (C) 2017 Lars Wirzenius This program is free software: you can redistribute it and/or modify @@ -17,22 +17,30 @@ set -eu +title() +{ + printf '\n%s\n' "$@" + for i in $(seq 77) + do + printf '%c' - + done + printf '\n' +} + run_yarns() { yarn yarns/*.yarn \ -s yarns/lib.py --shell python2 --shell-arg '' --cd-datadir "$@" } +title "Unit tests" python3 -m CoverageTestRunner --ignore-missing-from=without-tests -pep8 qvarn -pylint3 -j0 --rcfile pylint.conf qvarn - -if [ "${QVARN_POSTGRES:-no}" = no ] -then - # Run yarns without Postgres. For speed. - run_yarns "$@" -else - scripts/pgempty - run_yarns --env "QVARN_POSTGRES=$QVARN_POSTGRES" "$@" -fi +title "Code style" +pycodestyle salami + +title "Static checking" +pylint3 -j0 --rcfile pylint.conf salami + +title OK +echo All tests pass. diff --git a/debian/changelog b/debian/changelog index df2aea5..1f2cb16 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,22 +1,4 @@ -qvarn-jsonb (0.86+git-1) UNRELEASED; urgency=medium - - * New upstream version. - - -- Lars Wirzenius <liw@liw.fi> Tue, 10 Oct 2017 11:58:51 +0300 - -qvarn-jsonb (0.86-1) unstable; urgency=medium - - * New upstream version. - - -- Lars Wirzenius <liw@liw.fi> Tue, 10 Oct 2017 11:58:48 +0300 - -qvarn-jsonb (0.85-1) unstable; urgency=medium - - * New upstream version. - - -- Lars Wirzenius <liw@liw.fi> Wed, 27 Sep 2017 19:21:33 +0300 - -qvarn-jsonb (0.84-1) unstable; urgency=medium +salami (0.1-1) unstable; urgency=medium * Initial packaging. This is not intended to be uploaded to Debian, so no closing of an ITP bug. diff --git a/debian/control b/debian/control index 4b3ce7b..90c2f98 100644 --- a/debian/control +++ b/debian/control @@ -1,4 +1,4 @@ -Source: qvarn-jsonb +Source: salami Maintainer: QvarnLabs <info@qvarnlabs.com> Uploaders: Lars Wirzenius <liw@qvarnlabs.com> Section: web @@ -12,11 +12,9 @@ Build-Depends: debhelper (>= 9), python3-all, python3-apifw X-Python3-Version: >= 3.5 -Package: qvarn-jsonb +Package: salami Architecture: all Depends: ${python3:Depends}, ${misc:Depends}, python3 (>= 3.5) -Description: web service backend for structured data storage - This is a backend service for storing structured (JSON) and some - associated binary data using a REST-ful HTTP API and OpenID Connect - authentication. +Description: a thing + This is a thing. diff --git a/debian/copyright b/debian/copyright index 668a192..687e273 100644 --- a/debian/copyright +++ b/debian/copyright @@ -1,5 +1,5 @@ Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ -Upstream-Name: qvarn +Upstream-Name: salami Files: * Copyright: 2017, Lars Wirzenius diff --git a/debian/qvarn-jsonb.install b/debian/qvarn-jsonb.install deleted file mode 100644 index a8fe43f..0000000 --- a/debian/qvarn-jsonb.install +++ /dev/null @@ -1,2 +0,0 @@ -resource_type/*.yaml etc/qvarn/resource_type -start_qvarn usr/bin @@ -1,54 +0,0 @@ -#!/usr/bin/python3 - - -import qvarn - - -args = { - 'host': 'localhost', - 'port': 5432, - 'database': 'qvarn', - 'user': 'qvarn', - 'password': 'pass', - 'min_conn': 1, - 'max_conn': 1, -} - - -sql = qvarn.PostgresAdapter() -sql.connect(**args) - -pos = qvarn.PostgresObjectStore(sql) -pos.create_store(obj_id=str, revision=str) - - -objs = [ - { - 'foo': 'bar', - 'id': '1', - 'revision': '1.1', - }, - { - 'foo': 'yo', - 'id': '2', - 'revision': '2.1', - }, -] -for obj in objs: - pos.create_object(obj, obj_id=obj['id'], revision=obj['revision']) - -print('finding') -# cond = qvarn.Equal('foo', 'bar') -cond = qvarn.Yes() -objs = pos.get_objects(obj_id=obj['id']) -print(repr(objs)) -for obj in objs: - print('obj: ', repr(obj)) -print('done finding') - -print('finding keys') -cond = qvarn.Equal('foo', 'bar') -objs = pos.find_objects(cond) -for obj in objs: - print('keys: ', repr(obj)) -print('done finding keys') diff --git a/qvarn/__init__.py b/qvarn/__init__.py deleted file mode 100644 index 7d35b1a..0000000 --- a/qvarn/__init__.py +++ /dev/null @@ -1,114 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -from .version import __version__, __version_info__ -from .logging import log, setup_logging -from .idgen import ResourceIdGenerator -from .schema import schema -from .resource_type import ( - ResourceType, - load_resource_types, - add_missing_fields, -) -from .sql import ( - PostgresAdapter, - quote, - placeholder, - get_unique_name, - All, - Contains, - Equal, - GreaterThan, - GreaterOrEqual, - LessThan, - LessOrEqual, - NotEqual, - ResourceTypeIs, - Startswith, - Yes, -) -from .sql_select import sql_select, flatten - -from .objstore import ( - ObjectStoreInterface, - MemoryObjectStore, - PostgresObjectStore, - KeyCollision, - UnknownKey, - WrongKeyType, - KeyValueError, - NoSuchObject, - BlobKeyCollision, - flatten_object, -) - -from .validator import ( - Validator, - ValidationError, - NotADict, - NoType, - HasId, - HasRevision, - NoId, - NoRevision, - UnknownField, - UnknownSubpath, - WrongType, -) - -from .search_parser import ( - SearchParser, - SearchParameters, - SearchParserError, - NeedSortOperator, -) - -from .collection import ( - CollectionAPI, - NoSearchCriteria, - NoSuchResource, - UnknownSearchField, - WrongRevision, -) - -from .responses import ( - bad_request_response, - conflict_response, - created_response, - need_sort_response, - no_such_resource_response, - ok_response, - search_parser_error_response, - unknown_search_field_response, -) - -from .api_errors import ( - IdMismatch, - NoSuchResourceType, - NotJson, - TooManyResources, - TooManyResourceTypes, -) - -from .router import Router -from .file_router import FileRouter -from .notification_router import NotificationRouter -from .resource_router import ResourceRouter -from .subresource_router import SubresourceRouter -from .version_router import VersionRouter -from .timestamp import get_current_timestamp - -from .api import QvarnAPI diff --git a/qvarn/api.py b/qvarn/api.py deleted file mode 100644 index aea0887..0000000 --- a/qvarn/api.py +++ /dev/null @@ -1,187 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class QvarnAPI: - - def __init__(self): - self._store = None - self._validator = qvarn.Validator() - self._baseurl = None - self._rt_coll = None - - def set_base_url(self, baseurl): # pragma: no cover - self._baseurl = baseurl - - def set_object_store(self, store): - self._store = store - self._store.create_store(obj_id=str, subpath=str) - - def add_resource_type(self, rt): - path = rt.get_path() - keys = { - 'obj_id': rt.get_type(), - 'subpath': '', - } - self._store.remove_objects(**keys) - - obj = { - 'id': rt.get_type(), - 'type': 'resource_type', - 'path': path, - 'spec': rt.as_dict(), - } - self._store.create_object(obj, **keys, auxtable=True) - - def get_resource_type(self, path): - objs = self._get_resource_type_given_path(path) - if len(objs) == 0: - qvarn.log.log( - 'error', - msg_text='There is no resource type for path', - path=path) - raise qvarn.NoSuchResourceType(path) - elif len(objs) > 1: # pragma: no cover - qvarn.log.log( - 'error', - msg_text='There are more than one resource types for path', - path=path, - objs=objs) - raise qvarn.TooManyResourceTypes(path) - rt = qvarn.ResourceType() - rt.from_spec(objs[0]['spec']) - return rt - - def _get_resource_type_given_path(self, path): - cond = qvarn.All( - qvarn.Equal('path', path), - qvarn.Equal('type', 'resource_type'), - ) - results = self._store.find_objects(cond) - return [obj for _, obj in results] - - def get_listener_resource_type(self): - return self._get_resource_type_given_type('listener') - - def get_notification_resource_type(self): # pragma: no cover - return self._get_resource_type_given_type('notification') - - def _get_resource_type_given_type(self, type_name): - cond = qvarn.All( - qvarn.Equal('id', type_name), - qvarn.Equal('type', 'resource_type'), - ) - results = self._store.find_objects(cond) - objs = [obj for _, obj in results] - - if len(objs) == 0: # pragma: no cover - raise qvarn.NoSuchResourceType(type_name) - elif len(objs) > 1: # pragma: no cover - raise qvarn.TooManyResourceTypes(type_name) - - rt = qvarn.ResourceType() - rt.from_spec(objs[0]['spec']) - return rt - - def find_missing_route(self, path): - qvarn.log.log('info', msg_text='find_missing_route', path=path) - - if path == '/version': - qvarn.log.log('info', msg_text='Add /version route') - v = qvarn.VersionRouter() - return v.get_routes() - - try: - rt = self.get_resource_type(path) - except qvarn.NoSuchResourceType: - qvarn.log.log('warning', msg_text='No such route', path=path) - return [] - - routes = self.resource_routes(path, rt) - qvarn.log.log('info', msg_text='Found missing routes', routes=routes) - return routes - - def resource_routes(self, path, rt): # pragma: no cover - coll = qvarn.CollectionAPI() - coll.set_object_store(self._store) - coll.set_resource_type(rt) - - router = qvarn.ResourceRouter() - router.set_baseurl(self._baseurl) - router.set_collection(coll) - router.set_notifier(self.notify) - routes = router.get_routes() - - files = rt.get_files() - for subpath in rt.get_subpaths(): - if subpath not in files: - sub_router = qvarn.SubresourceRouter() - sub_router.set_subpath(subpath) - sub_router.set_parent_collection(coll) - more = sub_router.get_routes() - else: - file_router = qvarn.FileRouter() - file_router.set_subpath(subpath) - file_router.set_object_store(self._store) - file_router.set_parent_collection(coll) - more = file_router.get_routes() - routes.extend(more) - - listener_rt = self.get_listener_resource_type() - notif_router = qvarn.NotificationRouter() - notif_router.set_baseurl(self._baseurl) - notif_router.set_parent_collection(coll) - notif_router.set_object_store(self._store, listener_rt) - routes.extend(notif_router.get_routes()) - - return routes - - def notify(self, rid, rrev, change): # pragma: no cover - rt = self.get_notification_resource_type() - notifs = qvarn.CollectionAPI() - notifs.set_object_store(self._store) - notifs.set_resource_type(rt) - obj = { - 'type': 'notification', - 'resource_id': rid, - 'resource_revision': rrev, - 'resource_change': change, - 'timestamp': qvarn.get_current_timestamp(), - } - for listener in self.find_listeners(rid, change): - obj['listener_id'] = listener['id'] - qvarn.log.log( - 'info', msg_text='Notify listener of change', - notification=obj) - notifs.post(obj) - - def find_listeners(self, rid, change): # pragma: no cover - cond = qvarn.Equal('type', 'listener') - pairs = self._store.find_objects(cond) - for _, obj in pairs: - if self.listener_matches(obj, rid, change): - yield obj - - def listener_matches(self, obj, rid, change): # pragma: no cover - if change == 'created' and obj.get('notify_of_new'): - return True - if change != 'created' and obj.get('listen_on_all'): - return True - if rid in obj.get('listen_on', []): - return True - return False diff --git a/qvarn/api_errors.py b/qvarn/api_errors.py deleted file mode 100644 index 368ddc5..0000000 --- a/qvarn/api_errors.py +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -class NoSuchResourceType(Exception): # pragma: no cover - - def __init__(self, path): - super().__init__('No resource type for path {}'.format(path)) - - -class TooManyResourceTypes(Exception): # pragma: no cover - - def __init__(self, path): - super().__init__('Too many resource types for path {}'.format(path)) - - -class TooManyResources(Exception): # pragma: no cover - - def __init__(self, resource_id): - super().__init__('Too many resources with id {}'.format(resource_id)) - - -class NotJson(Exception): # pragma: no cover - - def __init__(self, ct): - super().__init__('Was expecting application/json, not {}'.format(ct)) - - -class IdMismatch(Exception): # pragma: no cover - - def __init__(self, obj_id, id_from_path): - super().__init__( - 'Resource has id {} but path says {}'.format(obj_id, id_from_path)) diff --git a/qvarn/api_tests.py b/qvarn/api_tests.py deleted file mode 100644 index b6f8929..0000000 --- a/qvarn/api_tests.py +++ /dev/null @@ -1,176 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import copy -import os -import unittest - - -import qvarn - - -class QvarnAPITests(unittest.TestCase): - - def test_returns_routes_for_version_path(self): - api = qvarn.QvarnAPI() - self.assertNotEqual(api.find_missing_route('/version'), []) - - def test_returns_no_routes_for_unknown_resource_type(self): - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - self.assertEqual(api.find_missing_route('/subjects'), []) - - def test_returns_routes_for_known_resource_type(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - }, - 'subpaths': { - 'sub': { - 'prototype': { - 'subfoo': '', - }, - }, - }, - }, - ], - } - - rt = qvarn.ResourceType() - rt.from_spec(spec) - - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - api.add_resource_type(rt) - api.set_base_url('https://qvarn.example.com') - - dirname = os.path.dirname(qvarn.__file__) - dirname = os.path.join(dirname, '../resource_type') - resource_types = qvarn.load_resource_types(dirname) - for rt in resource_types: - api.add_resource_type(rt) - - self.assertNotEqual(api.find_missing_route('/subjects'), []) - - def test_get_resource_type_raises_error_for_unknown_path(self): - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - with self.assertRaises(qvarn.NoSuchResourceType): - api.get_resource_type('/subjects') - - def test_get_resource_type_returns_it_when_it_is_known(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - }, - }, - ], - } - - rt = qvarn.ResourceType() - rt.from_spec(spec) - - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - api.add_resource_type(rt) - rt2 = api.get_resource_type(spec['path']) - self.assertTrue(isinstance(rt2, qvarn.ResourceType)) - self.assertEqual(rt2.as_dict(), spec) - - def test_add_resource_type_is_ok_adding_type_again(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - }, - }, - ], - } - - rt = qvarn.ResourceType() - rt.from_spec(spec) - - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - api.add_resource_type(rt) - self.assertEqual(api.add_resource_type(rt), None) - - def test_updating_resource_type_with_new_version_works(self): - spec1 = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - }, - }, - ], - } - - spec2 = copy.deepcopy(spec1) - spec2['versions'].append({ - 'version': 'v1', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - 'newfield': '', - }, - }) - - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - - rt1 = qvarn.ResourceType() - rt1.from_spec(spec1) - api.add_resource_type(rt1) - - rt2 = qvarn.ResourceType() - rt2.from_spec(spec2) - api.add_resource_type(rt2) - - rt = api.get_resource_type(spec1['path']) - self.assertEqual(rt.as_dict(), spec2) diff --git a/qvarn/backend.py b/qvarn/backend.py deleted file mode 100644 index 4a3eab8..0000000 --- a/qvarn/backend.py +++ /dev/null @@ -1,153 +0,0 @@ -#!/usr/bin/python3 -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -# This is a Qvarn backend program that serves all the API requests. It -# is based on apifw, which takes care of routing, logging, and -# authorization. - - -import os - -import yaml - -import apifw -import qvarn -import slog - - -DEFAULT_CONFIG_FILE = '/dev/null' - - -def dict_logger(log, stack_info=None): - qvarn.log.log(exc_info=stack_info, **log) - - -def read_config(filename): - with open(filename) as f: - return yaml.safe_load(f) - - -def check_config(cfg): - for key in cfg: - if cfg[key] is None: - raise Exception('Configration %s should not be None' % key) - - -_counter = slog.Counter() - - -def counter(): - new_context = 'HTTP transaction {}'.format(_counter.increment()) - qvarn.log.set_context(new_context) - - -default_config = { - 'baseurl': 'https://unconfigured-base-url/', - 'token-public-key': None, - 'token-audience': None, - 'token-issuer': None, - 'log': [], - 'resource-type-dir': None, - 'memory-database': True, - 'database': { - 'host': None, - 'port': 5432, - 'database': None, - 'user': None, - 'min_conn': 1, - 'max_conn': 1, - 'password': None, - }, -} - - -config_filename = os.environ.get('QVARN_CONFIG', DEFAULT_CONFIG_FILE) -actual_config = read_config(config_filename) -config = dict(default_config) -config.update(actual_config or {}) -check_config(config) -qvarn.setup_logging(config) -qvarn.log.log('info', msg_text='Qvarn backend starting') - -subject = qvarn.ResourceType() -subject.from_spec({ - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'type': '', - 'revision': '', - 'random_id': '', - 'names': [ - { - 'full_name': '', - 'sort_key': '', - 'titles': [''], - 'given_names': [''], - 'surnames': [''], - }, - ], - }, - 'subpaths': { - 'sub': { - 'prototype': { - 'subfield': '', - }, - }, - 'blob': { - 'prototype': { - 'body': 'blob', - 'content-type': '', - }, - }, - }, - 'files': [ - 'blob', - ], - }, - ], -}) - -resource_types = qvarn.load_resource_types(config['resource-type-dir']) - -if config['memory-database']: - store = qvarn.MemoryObjectStore() -else: - sql = qvarn.PostgresAdapter() - sql.connect(**config['database']) - store = qvarn.PostgresObjectStore(sql) - -api = qvarn.QvarnAPI() -api.set_base_url(config['baseurl']) -api.set_object_store(store) -api.add_resource_type(subject) -for rt in resource_types: - api.add_resource_type(rt) - -app = apifw.create_bottle_application( - api, counter, dict_logger, config, resource_types) - -# If we are running this program directly with Python, and not via -# gunicorn, we can use the Bottle built-in debug server, which can -# make some things easier to debug. - -if __name__ == '__main__': - print('running in debug mode') - app.run(host='127.0.0.1', port=12765) diff --git a/qvarn/collection.py b/qvarn/collection.py deleted file mode 100644 index 8a4f863..0000000 --- a/qvarn/collection.py +++ /dev/null @@ -1,300 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class CollectionAPI: - - object_keys = { - 'obj_id': str, - 'subpath': str, - } - - def __init__(self): - self._store = None - self._type = None - self._proto = None - self._idgen = qvarn.ResourceIdGenerator() - - def set_object_store(self, store): - self._store = store - store.create_store(**self.object_keys) - - def set_resource_type(self, rt): - assert isinstance(rt, qvarn.ResourceType) - self._type = rt - self._proto = self._type.get_latest_prototype() - - def get_type(self): - return self._type - - def get_type_name(self): - return self._type.get_type() - - def post(self, obj): - v = qvarn.Validator() - v.validate_new_resource(obj, self.get_type()) - - new_obj = self._new_object(self._proto, obj) - new_obj['id'] = self._invent_id(obj['type']) - new_obj['revision'] = self._invent_id('revision') - self._create_object(new_obj, obj_id=new_obj['id'], subpath='') - - rt = self.get_type() - subprotos = rt.get_subpaths() - for subpath, subproto in subprotos.items(): - empty = self._new_object(subproto, {}) - self._create_object(empty, obj_id=new_obj['id'], subpath=subpath) - - return new_obj - - def _create_object(self, obj, **keys): - assert set(keys.keys()) == set(self.object_keys.keys()) - qvarn.log.log( - 'debug', msg_text='Collection._create_object', obj=obj, keys=keys) - self._store.create_object(obj, **keys) - - def _new_object(self, proto, obj): - return qvarn.add_missing_fields(proto, obj) - - def _invent_id(self, resource_type): - return self._idgen.new_id(resource_type) - - def get(self, obj_id): - return self._get_object(obj_id, '') - - def get_subresource(self, obj_id, subpath): - return self._get_object(obj_id, subpath) - - def _get_object(self, obj_id, subpath): - keys = { - 'obj_id': obj_id, - 'subpath': subpath, - } - qvarn.log.log( - 'debug', msg_text='CollectionAPI._get_object', keys=keys) - objs = self._store.get_objects(**keys) - assert len(objs) <= 1 - if objs: - return objs[0] - raise NoSuchResource(**keys) - - def delete(self, obj_id): - self.get(obj_id) - self._store.remove_objects(obj_id=obj_id) - - def list(self): - oftype = qvarn.Equal('type', self.get_type_name()) - matches = self._store.find_objects(oftype) - qvarn.log.log('xxx', matches=matches, type=self.get_type_name()) - return { - 'resources': [ - {'id': obj['id']} - for _, obj in matches - if obj['type'] == self.get_type_name() - ] - } - - def put(self, obj): - v = qvarn.Validator() - v.validate_resource_update(obj, self.get_type()) - - old = self.get(obj['id']) - if old['revision'] != obj['revision']: - raise WrongRevision(obj['revision'], old['revision']) - - new_obj = dict(obj) - new_obj['revision'] = self._invent_id('revision') - self._store.remove_objects(obj_id=new_obj['id'], subpath='') - self._create_object(new_obj, obj_id=new_obj['id'], subpath='') - - return new_obj - - def put_subresource(self, sub_obj, subpath=None, **keys): - assert subpath is not None - obj_id = keys.pop('obj_id') - revision = keys.pop('revision') - parent = self.get(obj_id) - if parent['revision'] != revision: - raise WrongRevision(revision, parent['revision']) - keys = { - 'obj_id': obj_id, - 'subpath': subpath, - } - self._store.remove_objects(**keys) - self._create_object(sub_obj, **keys) - - parent = self._update_revision(obj_id) - new_sub = dict(sub_obj) - new_sub['revision'] = parent['revision'] - return new_sub - - def _update_revision(self, obj_id): - obj = self.get(obj_id) - obj['revision'] = self._invent_id('revision') - self._store.remove_objects(obj_id=obj_id, subpath='') - self._create_object(obj, obj_id=obj_id, subpath='') - return obj - - def search(self, search_criteria): - if not search_criteria: - raise NoSearchCriteria() - - p = qvarn.SearchParser() - sp = p.parse(search_criteria) - if sp.cond is None: - sp.cond = qvarn.Equal('type', self.get_type_name()) - - def pick_all(obj): - return obj - - def pick_id(obj): - return { - 'id': obj['id'], - } - - def pick_some_from_object(obj, fields): - return { - key: obj[key] - for key in obj - if key in fields - } - - def pick_some(fields): - return lambda obj: pick_some_from_object(obj, fields) - - if sp.show_all: - pick_fields = pick_all - elif sp.show_fields: - show_what = sp.show_fields + ['id'] - pick_fields = pick_some(show_what) - else: - pick_fields = pick_id - - # FIXME: This is needed because Qvarn API stupidly requires - # all fields to actually be defined by the resource type. If - # we drop that, we can drop this check, but that needs to be a - # managed transition, and for now we can't just drop it. - self._check_fields_are_allowed(sp.cond) - - unsorted = self._find_matches(sp.cond) - if sp.sort_keys: - result = self._sort_objects(unsorted, sp.sort_keys) - else: - result = unsorted - - if sp.offset is None and sp.limit is None: - chosen = result - elif sp.offset is None and sp.limit is not None: - chosen = result[:sp.limit] - elif sp.offset is not None and sp.limit is None: - chosen = result[sp.offset:] - elif sp.offset is not None and sp.limit is not None: - chosen = result[sp.offset:sp.offset+sp.limit] - - picked = [pick_fields(o) for o in chosen] - - qvarn.log.log( - 'trace', msg_text='Collection.search, sorted', - result=picked) - - return picked - - def _check_fields_are_allowed(self, cond): - names = set(self._get_names_from_cond(cond)) - allowed = set(self._get_allowed_names()) - for name in names.difference(allowed): - raise UnknownSearchField(name) - - def _get_names_from_cond(self, cond): - for c in qvarn.flatten(cond): - name = getattr(c, 'name') - if name is not None: - yield name - - def _get_allowed_names(self): - rt = self.get_type() - proto = rt.get_latest_prototype() - for name in self._get_names_from_prototype(proto): - yield name - - for subpath in rt.get_subpaths(): - subproto = rt.get_subprototype(subpath) - for name in self._get_names_from_prototype(subproto): - yield name - - def _get_names_from_prototype(self, proto): - schema = qvarn.schema(proto) - for t in schema: - for name in t[0]: - yield name - - def _find_matches(self, cond): - matches = self._store.find_objects(cond) - qvarn.log.log('xxx', matches=matches) - obj_ids = self._uniq(keys['obj_id'] for keys, _ in matches) - objects = [ - self._get_object(obj_id=obj_id, subpath='') - for obj_id in obj_ids - ] - return [o for o in objects if o['type'] == self.get_type_name()] - - def _uniq(self, items): - seen = set() - for item in items: - if item not in seen: - yield item - seen.add(item) - - def _sort_objects(self, objects, sort_keys): - def object_sort_key(obj, fields): - return [ - (key, value) - for key, value in qvarn.flatten_object(obj) - if key in fields - ] - - return sorted(objects, key=lambda o: object_sort_key(o, sort_keys)) - - -class WrongRevision(Exception): - - def __init__(self, actual, expected): - super().__init__( - 'PUTted objects must have correct revision set: ' - 'got {}, expected {}'.format(actual, expected)) - - -class NoSuchResource(Exception): - - def __init__(self, **keys): - keys_str = ', '.join( - '{}={!r}'.format(key, keys[key]) for key in sorted(keys)) - super().__init__("There is no resource with keys {}".format(keys_str)) - - -class UnknownSearchField(Exception): - - def __init__(self, field): - super().__init__('There is no field {}'.format(field)) - self.field = field - - -class NoSearchCriteria(Exception): - - def __init__(self): - super().__init__('No search criteria was given') diff --git a/qvarn/collection_tests.py b/qvarn/collection_tests.py deleted file mode 100644 index abc776e..0000000 --- a/qvarn/collection_tests.py +++ /dev/null @@ -1,448 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import itertools -import unittest - -import qvarn - - -class CollectionAPITests(unittest.TestCase): - - def setUp(self): - self.store = qvarn.MemoryObjectStore() - - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'type': '', - 'id': '', - 'revision': '', - 'full_name': '', - 'names': [ - { - 'sort_key': '', - }, - ], - 'things': [ - { - 'things': '', - 'other': '', - }, - ], - }, - 'subpaths': { - 'sub': { - 'prototype': { - 'subfield': '', - }, - }, - }, - }, - ], - } - self.rt = qvarn.ResourceType() - self.rt.from_spec(spec) - - self.coll = qvarn.CollectionAPI() - self.coll.set_object_store(self.store) - self.coll.set_resource_type(self.rt) - - def test_returns_specified_type(self): - self.assertEqual(self.coll.get_type(), self.rt) - - def test_returns_specified_type_name(self): - self.assertEqual(self.coll.get_type_name(), self.rt.get_type()) - - def test_post_raises_error_if_type_not_given(self): - obj = { - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.NoType): - self.coll.post(obj) - - def test_post_raises_error_if_type_is_not_expected_one(self): - obj = { - 'type': 'unperson', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.WrongType): - self.coll.post(obj) - - def test_post_raises_error_if_id_given(self): - obj = { - 'id': 'object-1', - 'type': 'subject', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.HasId): - self.coll.post(obj) - - def test_post_raises_error_if_revision_given(self): - obj = { - 'revision': 'rev-1', - 'type': 'subject', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.HasRevision): - self.coll.post(obj) - - def test_post_creates_a_new_resource(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - self.assertTrue(new_obj['id']) - self.assertTrue(new_obj['revision']) - self.assertEqual(new_obj['things'], []) - self.assertEqual(new_obj, self.coll.get(new_obj['id'])) - - sub = self.coll.get_subresource(new_obj['id'], 'sub') - self.assertEqual(sub, {'subfield': ''}) - - def test_post_creates_a_new_resource_with_dict_list(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - 'things': [ - { - 'other': 'foo', - }, - ], - } - new_obj = self.coll.post(obj) - self.assertTrue(new_obj['id']) - self.assertTrue(new_obj['revision']) - self.assertEqual( - new_obj['things'], - [ - { - 'things': '', - 'other': 'foo', - }, - ] - ) - self.assertEqual(new_obj, self.coll.get(new_obj['id'])) - - def test_post_creates_a_new_id_revision_every_time(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj1 = self.coll.post(obj) - new_obj2 = self.coll.post(obj) - self.assertNotEqual(new_obj1, new_obj2) - - def test_get_raise_error_if_not_found(self): - with self.assertRaises(qvarn.NoSuchResource): - self.coll.get('no-such-object-id') - - def test_deleting_nonexisent_resource_raise_error(self): - with self.assertRaises(qvarn.NoSuchResource): - self.coll.delete('no-such-object-id') - - def test_deleting_resource_makes_it_go_away(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - self.coll.delete(new_obj['id']) - with self.assertRaises(qvarn.NoSuchResource): - self.coll.get(new_obj['id']) - - def test_listing_objects_returns_empty_list_initially(self): - self.assertEqual(self.coll.list(), {'resources': []}) - - def test_listing_objects_returns_list_of_existing_object(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - - # Add an object of a different type so that we can make sure - # only objects of the right type are returned. - wrong_type = { - 'type': 'unperson', - 'full_name': 'James Bond', - } - self.store.create_object(wrong_type, obj_id='007') - - self.assertEqual( - self.coll.list(), - { - 'resources': [ - { - 'id': new_obj['id'] - } - ] - } - ) - - def test_putting_resource_without_id_raises_error(self): - obj = { - 'revision': 'revision-1', - 'type': 'subject', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.NoId): - self.coll.put(obj) - - def test_putting_resource_without_type_raises_error(self): - obj = { - 'id': 'object-id-1', - 'revision': 'revision-1', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.NoType): - self.coll.put(obj) - - def test_putting_resource_with_wrong_type_raises_error(self): - obj = { - 'id': 'object-id-1', - 'type': 'notasubject', - 'revision': 'revision-1', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.WrongType): - self.coll.put(obj) - - def test_putting_resource_without_revision_raises_error(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - del new_obj['revision'] - with self.assertRaises(qvarn.NoRevision): - self.coll.put(new_obj) - - def test_putting_resource_with_wrong_revision_raises_error(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - obj2 = dict(new_obj) - obj2['revision'] = 'this-revision-never-happens-randomly' - with self.assertRaises(qvarn.WrongRevision): - self.coll.put(obj2) - - def test_putting_nonexistent_resource_raises_error(self): - obj = { - 'id': 'object-id-1', - 'revision': 'revision-1', - 'type': 'subject', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.NoSuchResource): - self.coll.put(obj) - - def test_putting_resource_updates_resource(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - obj2 = dict(new_obj) - obj2['full_name'] = 'Alfred Newman' - newer_obj = self.coll.put(obj2) - self.assertNotEqual(new_obj['revision'], newer_obj['revision']) - self.assertEqual(self.revisionless(obj2), self.revisionless(newer_obj)) - - def revisionless(self, obj): - return { - key: value - for key, value in obj.items() - if key != 'revision' - } - - def test_putting_subresource(self): - parent = { - 'type': 'subject', - 'full_name': 'James Bond', - } - sub = { - 'subfield': 'subvalue', - } - new_obj = self.coll.post(parent) - obj_id = new_obj['id'] - revision = new_obj['revision'] - new_sub = self.coll.put_subresource( - sub, subpath='sub', obj_id=obj_id, revision=revision) - self.assertNotEqual(new_sub['revision'], revision) - self.assertEqual(self.coll.get_subresource(obj_id, 'sub'), sub) - - new_parent = self.coll.get(obj_id) - self.assertNotEqual(new_parent['revision'], revision) - self.assertEqual(new_parent['revision'], new_sub['revision']) - - def test_putting_subresource_raises_error_without_parent_object(self): - sub = { - 'subfield': 'subvalue', - } - with self.assertRaises(qvarn.NoSuchResource): - self.coll.put_subresource( - sub, obj_id='unknown', revision='unknown', subpath='sub') - - def test_putting_subresource_raises_error_if_revision_is_wrong(self): - parent = { - 'type': 'subject', - 'full_name': 'James Bond', - } - sub = { - 'subfield': 'subvalue', - } - new_obj = self.coll.post(parent) - with self.assertRaises(qvarn.WrongRevision): - self.coll.put_subresource( - sub, obj_id=new_obj['id'], revision='wrong', subpath='sub') - - def test_search_with_empty_criteria_raises_error(self): - with self.assertRaises(qvarn.NoSearchCriteria): - self.coll.search('') - - def test_search_with_comparison_on_unknown_field_raises_error(self): - with self.assertRaises(qvarn.UnknownSearchField): - self.coll.search('exact/DOESNOTEXIST/sanity') - - def test_search_without_matches_returns_empty_list(self): - self.assertEqual(self.coll.search('exact/full_name/nomatch'), []) - - def test_search_return_matching_resource_ids(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - matches = self.coll.search('exact/full_name/James Bond') - self.assertEqual(matches, [{'id': new_obj['id']}]) - - def test_search_ignores_case(self): - obj = { - 'type': 'subject', - 'full_name': 'JAMES', - } - new_obj = self.coll.post(obj) - matches = self.coll.search('exact/full_name/james') - self.assertEqual(matches, [{'id': new_obj['id']}]) - - def test_search_return_matching_resources_themselves(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - wanted = { - 'id': new_obj['id'], - 'full_name': new_obj['full_name'], - } - matches = self.coll.search('exact/full_name/James Bond/show/full_name') - self.assertEqual(matches, [wanted]) - - def test_search_return_matching_resources_if_subresource_matches(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - sub = { - 'subfield': 'xyzzy', - } - - new_obj = self.coll.post(obj) - self.coll.put_subresource( - sub, subpath='sub', obj_id=new_obj['id'], - revision=new_obj['revision']) - matches = self.coll.search('exact/full_name/James Bond/show_all') - self.assertEqual(matches, [new_obj]) - - def test_search_return_full_resources(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - matches = self.coll.search('exact/full_name/James Bond/show_all') - self.assertEqual(matches, [new_obj]) - - def test_search_sorts(self): - names = ['Alfred Pennyweather', 'James Bond', 'Jason Bourne'] - for perm in itertools.permutations(names): - self.setUp() - for name in perm: - obj = { - 'type': 'subject', - 'full_name': 'Spy', - 'names': [ - { - 'sort_key': name, - }, - ], - } - self.coll.post(obj) - - matches = self.coll.search( - 'exact/full_name/Spy/show_all/sort/sort_key') - self.assertEqual(len(matches), 3) - self.assertEqual( - [m['names'][0]['sort_key'] for m in matches], - names) - - def test_limit_without_sort(self): - with self.assertRaises(qvarn.SearchParserError): - self.coll.search('/limit/1') - - def test_offset_without_sort(self): - with self.assertRaises(qvarn.SearchParserError): - self.coll.search('/offset/1') - - def test_search_with_limit_only(self): - objs = self.create_objects(['1', '2', '3']) - search = 'sort/full_name/show_all' - matches = self.coll.search(search + '/limit/2') - self.assertEqual(len(matches), 2) - self.assertEqual(matches, objs[:2]) - - def test_search_with_offset_only(self): - objs = self.create_objects(['1', '2', '3']) - search = 'sort/full_name/show_all' - matches = self.coll.search(search + '/offset/2') - self.assertEqual(len(matches), 1) - self.assertEqual(matches, objs[2:]) - - def test_search_with_offset_and_limit(self): - objs = self.create_objects(['1', '2', '3']) - search = 'sort/full_name/show_all' - matches = self.coll.search(search + '/offset/1/limit/1') - self.assertEqual(len(matches), 1) - self.assertEqual(matches, objs[1:2]) - - def create_objects(self, names): - objs = [] - for name in names: - obj = { - 'type': 'subject', - 'full_name': name, - } - new_obj = self.coll.post(obj) - objs.append(new_obj) - return objs diff --git a/qvarn/file_router.py b/qvarn/file_router.py deleted file mode 100644 index b7e4507..0000000 --- a/qvarn/file_router.py +++ /dev/null @@ -1,98 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class FileRouter(qvarn.Router): - - def __init__(self): - super().__init__() - self._store = None - self._parent_coll = None - self._subpath = None - - def set_subpath(self, subpath): - self._subpath = subpath - - def set_parent_collection(self, parent_coll): - self._parent_coll = parent_coll - - def set_object_store(self, store): - self._store = store - - def get_routes(self): - rt = self._parent_coll.get_type() - file_path = '{}/<id>/{}'.format(rt.get_path(), self._subpath) - return [ - { - 'method': 'GET', - 'path': file_path, - 'callback': self._get_file, - }, - { - 'method': 'PUT', - 'path': file_path, - 'callback': self._put_file, - }, - ] - - def _get_file(self, *args, **kwargs): - obj_id = kwargs['id'] - try: - obj = self._parent_coll.get(obj_id) - sub_obj = self._parent_coll.get_subresource(obj_id, self._subpath) - blob = self._store.get_blob(obj_id=obj_id, subpath=self._subpath) - except (qvarn.NoSuchResource, qvarn.NoSuchObject) as e: - return qvarn.no_such_resource_response(str(e)) - headers = { - 'Content-Type': sub_obj['content_type'], - 'Revision': obj['revision'], - } - return qvarn.ok_response(blob, headers) - - def _put_file(self, content_type, body, *args, **kwargs): - obj_id = kwargs['id'] - - # FIXME: add header getting to apifw - import bottle - revision = bottle.request.get_header('Revision') - - obj = self._parent_coll.get(obj_id) - if obj['revision'] != revision: - qvarn.log.log( - 'error', - msg_text='Client gave wrong revision', - revision_from_client=revision, - current_revision=obj['revision']) - return qvarn.conflict_response( - 'Bad revision {}'.format(revision)) - - sub_obj = self._parent_coll.get_subresource(obj_id, self._subpath) - sub_obj['content_type'] = content_type - new_sub = self._parent_coll.put_subresource( - sub_obj, subpath=self._subpath, obj_id=obj_id, revision=revision) - - try: - self._store.remove_blob(obj_id=obj_id, subpath=self._subpath) - self._store.create_blob(body, obj_id=obj_id, subpath=self._subpath) - except qvarn.NoSuchObject as e: - return qvarn.no_such_resource_response(str(e)) - - headers = { - 'Revision': new_sub['revision'], - } - return qvarn.ok_response('', headers) diff --git a/qvarn/idgen.py b/qvarn/idgen.py deleted file mode 100644 index f22332f..0000000 --- a/qvarn/idgen.py +++ /dev/null @@ -1,147 +0,0 @@ -# idgen.py - generate resource identifiers -# -# Copyright 2015, 2016 Suomen Tilaajavastuu Oy -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -# Implementation discussion -# ========================= -# -# A resource identifier must be unique, efficient, secure, and -# private. These not in priority order: all requirements must be -# fulfilled. Additionally, it would be good for identifiers to be -# practical. -# -# Uniqueness and efficiency -# ------------------------- -# -# Uniqueness can be provided by having a central database in which all -# identifiers exist, and adding new ones there. This is a performance -# bottleneck. We choose to rely on good, large random numbers instead. -# Quoting Wikipedia on random UUIDs: -# -# To put these numbers into perspective, the annual risk of a -# given person being hit by a meteorite is estimated to be one -# chance in 17 billion,[4] which means the probability is about -# 0.00000000006 (6 * 10**11), equivalent to the odds of creating -# a few tens of trillions of UUIDs in a year and having one -# duplicate. In other words, only after generating 1 billion -# UUIDs every second for the next 100 years, the probability of -# creating just one duplicate would be about 50%. -# -# See http://en.wikipedia.org/wiki/Universally_unique_identifier. -# -# We do not used the UUID format specifically, but the principle -# holds. If get random numbers from a high-quality source, and use at -# least 128 bits of randomness (UUID4 uses 122), we're good. A bug in -# other parts of our software, the database software, the operating -# system, hardware, or operational procedures is more likely to cause -# duplicates. -# -# Security and privacy -# -------------------- -# -# We want our identifiers to avoid leaking information. A linear -# counter, for example, would leak information: the number of resource -# objects created. We also can't use external identifiers, such as -# social security identifiers, as parts of resource identifiers, as -# this would leak actual sensitive information. Using random numbers -# is perfect. -# -# For added privacy, it would be good if different API clients would -# get different identifiers for the same resource. This would make it -# more difficult for them to combine data and endanger people's -# privacy. This level of protection is currently not handled at all, -# and in any case will probably not handled by this class. Instead, if -# we want to do this, we'll add a translation layer that changes -# internal identifiers to be per-client ones, and back again. -# -# Error checking -# -------------- -# -# Inevitably, resource identifiers will show up in URLs, in log files, -# and be communicated by humans using writing or voice. To make these -# things easier, it would be good to be able to verify that an -# identifier looks correct, by adding error checking into the identifier. -# -# Further, it would be good to have type information: is this -# identifier one for a person or an organisation? This can catch -# attempts at using an identifier for the wrong type of resource. -# -# Identifier strucure -# ------------------- -# -# Based on the above discussion, we define the following structure for -# a resource identifier: -# -# * 16 bits of type field -# * 128 bits of randomness -# * 32 bits of error checking -# -# The type field is the top 16 bits of a SHA-512 of the resources type: -# effectively this: -# -# hashlib.sha512('person').hexdigest()[:4] -# -# The random bits are read directly from /dev/urandom. Python provides -# os.urandom and uuid.uuid4, which could either be used, but to avoid -# having to trust Python's implementation, we read /dev/urandom -# directly. -# -# The error checking is done by computing the SHA-512 of the rest of -# the identifier and taking the top 32 bits: -# -# hashlib.sha512(rest).hexdigest()[:8] -# -# For human convenience, we allow identifiers (which are effectively -# very large hexadecimal numbers) to be represented in upper or lower -# case, and that any non-hexdigits are ignored. -# -# The canonical form of an identifier (in this case, for a person): -# -# 0035-94c4f55599453307002f0731e0b67999-9ffa4cf4 - - -import hashlib -import os - - -class ResourceIdGenerator: - - '''Generate resource identifiers and revisions.''' - - def new_id(self, resource_type): - '''Generate a new identifier.''' - - type_field = self._encode_type(resource_type) - random_field = self._get_randomness() - checksum_field = self._compute_checksum(type_field + random_field) - return self._canonical_form(type_field, random_field, checksum_field) - - def _encode_type(self, resource_type): - return hashlib.sha512(resource_type.encode('ascii')).hexdigest()[:4] - - def _get_randomness(self): - num_bits = 128 - num_bytes = num_bits // 8 - random_bytes = os.urandom(num_bytes) - return random_bytes.hex() - - def _compute_checksum(self, rest): - return hashlib.sha512(rest.encode('ascii')).hexdigest()[:8] - - def _canonical_form(self, type_field, random_field, checksum_field): - return '{0}-{1}-{2}'.format(type_field, random_field, checksum_field) diff --git a/qvarn/idgen_tests.py b/qvarn/idgen_tests.py deleted file mode 100644 index 944db1c..0000000 --- a/qvarn/idgen_tests.py +++ /dev/null @@ -1,36 +0,0 @@ -# idgen_tests.py - unit tests for ResourceIdGenerator -# -# Copyright 2015, 2016 Suomen Tilaajavastuu Oy -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - -import qvarn - - -class ResourceIdGeneratorTests(unittest.TestCase): - - def test_returns_a_unicode_string(self): - rig = qvarn.ResourceIdGenerator() - resource_id = rig.new_id('person') - self.assertTrue(isinstance(resource_id, str)) - - def test_returns_new_values_each_time(self): - rig = qvarn.ResourceIdGenerator() - id_1 = rig.new_id('person') - id_2 = rig.new_id('person') - self.assertNotEqual(id_1, id_2) diff --git a/qvarn/notification_router.py b/qvarn/notification_router.py deleted file mode 100644 index b96e38e..0000000 --- a/qvarn/notification_router.py +++ /dev/null @@ -1,254 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class NotificationRouter(qvarn.Router): - - def __init__(self): - super().__init__() - self._baseurl = None - self._store = None - self._parent_coll = None - self._listener_coll = None - - def set_baseurl(self, baseurl): - self._baseurl = baseurl - - def set_parent_collection(self, parent_coll): - self._parent_coll = parent_coll - - def set_object_store(self, store, listener_rt): - self._store = store - listeners = qvarn.CollectionAPI() - listeners.set_object_store(self._store) - listeners.set_resource_type(listener_rt) - self._listener_coll = listeners - - def get_routes(self): - rt = self._parent_coll.get_type() - listeners_path = '{}/listeners'.format(rt.get_path()) - listener_id_path = '{}/<listener_id>'.format(listeners_path) - notifications_path = '{}/notifications'.format(listener_id_path) - notification_id_path = '{}/<notification_id>'.format( - notifications_path) - - return [ - { - 'method': 'POST', - 'path': listeners_path, - 'callback': self._create_listener, - }, - { - 'method': 'GET', - 'path': listeners_path, - 'callback': self._get_listener_list, - }, - { - 'method': 'GET', - 'path': listener_id_path, - 'callback': self._get_a_listener, - }, - { - 'method': 'PUT', - 'path': listener_id_path, - 'callback': self._update_listener, - }, - { - 'method': 'DELETE', - 'path': listener_id_path, - 'callback': self._delete_listener, - }, - { - 'method': 'GET', - 'path': notifications_path, - 'callback': self._get_notifications_list, - }, - { - 'method': 'GET', - 'path': notification_id_path, - 'callback': self._get_a_notification, - }, - { - 'method': 'DELETE', - 'path': notification_id_path, - 'callback': self._delete_notification, - }, - ] - - def _create_listener(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - rt = self._listener_coll.get_type() - validator = qvarn.Validator() - try: - validator.validate_against_prototype( - rt.get_type(), body, rt.get_latest_prototype()) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - if 'type' not in body: - body['type'] = 'listener' - - rtype = self._parent_coll.get_type_name() - if body.get('listen_on_type', rtype) != rtype: - return qvarn.bad_request_response( - 'listen_on_type does not have value {}'.format(rtype)) - body['listen_on_type'] = rtype - - result_body = self._listener_coll.post(body) - location = self._get_new_resource_location(result_body) - qvarn.log.log( - 'debug', msg_text='POST a new listener, result', - body=result_body, location=location) - return qvarn.created_response(result_body, location) - - def _get_new_resource_location(self, resource): - return '{}{}/listeners/{}'.format( - self._baseurl, self._parent_coll.get_type().get_path(), - resource['id']) - - def _get_listener_list(self, content_type, body, *args, **kwargs): - rtype = self._parent_coll.get_type_name() - listener_list = self._listener_coll.list()['resources'] - listener_ids = [listener['id'] for listener in listener_list] - listeners = [self._listener_coll.get(lid) for lid in listener_ids] - qvarn.log.log('trace', msg_text='xxx', listeners=listeners) - correct_ids = [ - {"id": listener['id']} - for listener in listeners - if listener['listen_on_type'] == rtype - ] - body = { - 'resources': correct_ids, - } - return qvarn.ok_response(body) - - def _get_a_listener(self, *args, **kwargs): - try: - obj = self._listener_coll.get(kwargs['listener_id']) - except qvarn.NoSuchResource as e: - return qvarn.no_such_resource_response(str(e)) - return qvarn.ok_response(obj) - - def _update_listener(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - if 'type' not in body: - body['type'] = 'listener' - - listener_id = kwargs['listener_id'] - if 'id' not in body: - body['id'] = listener_id - - validator = qvarn.Validator() - try: - validator.validate_resource_update( - body, self._listener_coll.get_type()) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - try: - result_body = self._listener_coll.put(body) - except qvarn.WrongRevision as e: - return qvarn.conflict_response(str(e)) - except qvarn.NoSuchResource as e: - # We intentionally say bad request, instead of not found. - # This is to be compatible with old Qvarn. This may get - # changed later. - return qvarn.bad_request_response(str(e)) - - return qvarn.ok_response(result_body) - - def _delete_listener(self, *args, **kwargs): - listener_id = kwargs['listener_id'] - self._listener_coll.delete(listener_id) - for obj_id in self._find_notifications(listener_id): - self._store.remove_objects(obj_id=obj_id) - return qvarn.ok_response({}) - - def _find_notifications(self, listener_id): - cond = qvarn.All( - qvarn.Equal('type', 'notification'), - qvarn.Equal('listener_id', listener_id), - ) - obj_ids = [ - keys['obj_id'] - for keys, _ in self._store.find_objects(cond) - ] - qvarn.log.log( - 'trace', msg_text='Found notifications', - notifications=obj_ids) - return obj_ids - - def _get_notifications_list(self, *args, **kwargs): - def timestamp(pair): - _, obj = pair - return obj['timestamp'] - - listener_id = kwargs['listener_id'] - cond = qvarn.All( - qvarn.Equal('type', 'notification'), - qvarn.Equal('listener_id', listener_id) - ) - pairs = self._store.find_objects(cond) - ordered = sorted(pairs, key=timestamp) - body = { - 'resources': [ - { - 'id': keys['obj_id'] - } - for keys, _ in ordered - ] - } - return qvarn.ok_response(body) - - def _get_a_notification(self, *args, **kwargs): - listener_id = kwargs['listener_id'] - notification_id = kwargs['notification_id'] - cond = qvarn.All( - qvarn.Equal('type', 'notification'), - qvarn.Equal('listener_id', listener_id), - qvarn.Equal('id', notification_id), - ) - pairs = self._store.find_objects(cond) - if len(pairs) == 0: - return qvarn.no_such_resource_response(notification_id) - if len(pairs) > 1: - raise qvarn.TooManyResources(notification_id) - return qvarn.ok_response(pairs[0][1]) - - def _delete_notification(self, *args, **kwargs): - listener_id = kwargs['listener_id'] - notification_id = kwargs['notification_id'] - cond = qvarn.All( - qvarn.Equal('type', 'notification'), - qvarn.Equal('listener_id', listener_id), - qvarn.Equal('id', notification_id), - ) - for keys, _ in self._store.find_objects(cond): - values = { - key: keys[key] - for key in keys - if isinstance(keys[key], str) - } - self._store.remove_objects(**values) - return qvarn.ok_response({}) diff --git a/qvarn/objstore.py b/qvarn/objstore.py deleted file mode 100644 index d88774e..0000000 --- a/qvarn/objstore.py +++ /dev/null @@ -1,413 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import json - - -import qvarn - - -class ObjectStoreInterface: # pragma: no cover - - '''Store and retrieve JSON-like objects. - - A JSON-like object is a Python dict whose keys are strings, and - values corresponding to they keys can serialised into JSON, so - they're strings, integers, booleans, or JSON-like objects, or - lists of such values. JSON would support more types, but Qvarn - doesn't need them. Value strings (but not key names) may be - Unicode text or binary strings. - - The object store stores the JSON-like object, and a set of keys - that identify the object. The caller gets to define the keys. The - keys must be strings. There can be any number of keys, but there - must be at least one. The caller gets to define the keys and their - meaning. The allowed keys (and their types) are set when the store - is created, using the create_store method. - - Objects may be retrieved or removed using any subset of keys. All - maching objects are retrieved or removed. - - Objects mey be found using conditions, implemented by subclasses - of the qvarn.Condition class. Various condidions may be combined - arbitrarily. - - This class is for defining the ObjectStore interface. There is an - in-memory variant, for use in unit tests, and a version using - PostgreSQL for production use. - - ''' - - def create_store(self, **keys): - raise NotImplementedError() - - def check_keys_have_str_type(self, **keys): - for key in keys: - if keys[key] != str: - raise WrongKeyType(key, keys[key]) - - def get_known_keys(self): - raise NotImplementedError() - - def check_all_keys_are_allowed(self, **keys): - known_keys = self.get_known_keys() - for key in keys: - if key not in known_keys: - raise UnknownKey(key) - - def check_value_types(self, **keys): - known_keys = self.get_known_keys() - for key in keys: - if type(keys[key]) is not known_keys[key]: - raise KeyValueError(key, keys[key]) - - def create_object(self, obj, auxtable=True, **keys): - raise NotImplementedError() - - def remove_objects(self, **keys): - raise NotImplementedError() - - def get_objects(self, **keys): - raise NotImplementedError() - - def find_objects(self, cond): - raise NotImplementedError() - - def create_blob(self, blob, subpath=None, **keys): - raise NotImplementedError() - - def get_blob(self, subpath=None, **keys): - raise NotImplementedError() - - def remove_blob(self, blob, subpath=None, **keys): - raise NotImplementedError() - - -class MemoryObjectStore(ObjectStoreInterface): - - def __init__(self): - self._objs = [] - self._blobs = [] - self._known_keys = {} - - def get_known_keys(self): - return self._known_keys - - def create_store(self, **keys): - self.check_keys_have_str_type(**keys) - qvarn.log.log( - 'trace', msg_text='Creating store', keys=repr(keys), exc_info=True) - self._known_keys = keys - - def create_object(self, obj, auxtable=True, **keys): - qvarn.log.log( - 'trace', msg_text='Creating object', object=repr(obj), keys=keys) - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - self._check_unique_object(**keys) - self._objs.append((obj, keys)) - - def _check_unique_object(self, **keys): - for _, k in self._objs: - if self._keys_match(k, keys): - raise KeyCollision(k) - - def create_blob(self, blob, **keys): - qvarn.log.log('trace', msg_text='Creating blob', keys=keys) - subpath = keys.pop('subpath') - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - self._check_unique_blob(subpath, **keys) - if not self.get_objects(**keys): - raise NoSuchObject(keys) - self._blobs.append((blob, subpath, keys)) - - def _check_unique_blob(self, subpath, **keys): - for _, s, k in self._blobs: - if self._keys_match(k, keys) and s == subpath: - raise BlobKeyCollision(subpath, k) - - def get_blob(self, **keys): - subpath = keys.pop('subpath') - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - blobs = [ - b - for b, s, k in self._blobs - if self._keys_match(k, keys) and s == subpath - ] - assert len(blobs) <= 1 - if not blobs: - raise NoSuchObject(keys) - return blobs[0] - - def remove_blob(self, **keys): - subpath = keys.pop('subpath') - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - self._blobs = [ - b - for b, s, k in self._blobs - if not self._keys_match(k, keys) or s != subpath - ] - - def remove_objects(self, **keys): - self.check_all_keys_are_allowed(**keys) - self._objs = [ - (o, k) for o, k in self._objs if not self._keys_match(k, keys)] - - def get_objects(self, **keys): - self.check_all_keys_are_allowed(**keys) - return [o for o, k in self._objs if self._keys_match(k, keys)] - - def _keys_match(self, got_keys, wanted_keys): - for key in wanted_keys.keys(): - if got_keys.get(key) != wanted_keys[key]: - return False - return True - - def find_objects(self, cond): - return [(keys, obj) for obj, keys in self._objs if cond.matches(obj)] - - -class PostgresObjectStore(ObjectStoreInterface): # pragma: no cover - - _table = '_objects' - _auxtable = '_aux' - _blobtable = '_blobs' - - def __init__(self, sql): - self._sql = sql - self._keys = None - - def get_known_keys(self): - return self._keys - - def create_store(self, **keys): - self.check_keys_have_str_type(**keys) - self._keys = dict(keys) - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.create_store', - keys=repr(keys)) - - # Create main table for objects. - self._create_table(self._table, self._keys, '_obj', dict) - - # Create helper table for fields at all depths. Needed by searches. - self._create_table(self._auxtable, self._keys, '_field', dict) - - # Create helper table for blobs. - self._create_table(self._blobtable, self._keys, '_blob', bytes) - - def _create_table(self, name, col_dict, col_name, col_type): - columns = dict(col_dict) - columns[col_name] = col_type - with self._sql.transaction() as t: - query = t.create_table(name, **columns) - t.execute(query, {}) - - def create_object(self, obj, auxtable=True, **keys): - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.create_object', - obj=obj, keys=keys) - with self._sql.transaction() as t: - self._remove_objects_in_transaction(t, **keys) - self._insert_into_object_table(t, self._table, obj, **keys) - if auxtable: - self._insert_into_helper(t, self._auxtable, obj, **keys) - - def _insert_into_object_table(self, t, table_name, obj, **keys): - keys['_obj'] = json.dumps(obj) - column_names = list(keys.keys()) - query = t.insert_object(table_name, *column_names) - t.execute(query, keys) - - def _insert_into_helper(self, t, table_name, obj, **keys): - for field, value in flatten_object(obj): - x = { - 'name': field, - 'value': value, - } - keys['_field'] = json.dumps(x) - column_names = list(keys.keys()) - query = t.insert_object(table_name, *column_names) - t.execute(query, keys) - - def remove_objects(self, **keys): - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.remove_objects', - keys=keys) - with self._sql.transaction() as t: - query = t.remove_objects(self._table, *keys.keys()) - t.execute(query, keys) - - query = t.remove_objects(self._auxtable, *keys.keys()) - t.execute(query, keys) - - def _remove_objects_in_transaction(self, t, **keys): - qvarn.log.log( - 'info', - msg_text='PostgresObjectStore._remove_objects_in_transaction', - keys=keys) - query = t.remove_objects(self._table, *keys.keys()) - t.execute(query, keys) - query = t.remove_objects(self._auxtable, *keys.keys()) - t.execute(query, keys) - - def get_objects(self, **keys): - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.get_objects', - keys=keys) - with self._sql.transaction() as t: - return self._get_objects_in_transaction(t, **keys) - - def _get_objects_in_transaction(self, t, **keys): - query = t.select_objects(self._table, '_obj', *keys.keys()) - qvarn.log.log( - 'debug', msg_text='PostgresObjectStore.get_objects', - query=query, keys=keys) - cursor = t.execute(query, keys) - return [row['_obj'] for row in t.get_rows(cursor)] - - def find_objects(self, cond): - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.find_objects', - cond=repr(cond)) - with self._sql.transaction() as t: - rows = self._find_helper(t, cond) - return [ - self._split_row(row) - for row in rows - if row['subpath'] == '' - ] - - def _find_helper(self, t, cond): - keys_columns = [key for key in self._keys if key != '_obj'] - query, values = t.select_objects_on_cond( - self._auxtable, cond, *keys_columns) - cursor = t.execute(query, values) - return t.get_rows(cursor) - - def _split_row(self, row): - keys = dict(row) - obj = row.pop('_obj') - return keys, obj - - def create_blob(self, blob, **keys): - qvarn.log.log('trace', msg_text='Creating blob', keys=keys) - - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - if not self.get_objects(**keys): - raise NoSuchObject(keys) - - with self._sql.transaction() as t: - column_names = list(keys.keys()) + ['_blob'] - query = t.insert_object(self._blobtable, *column_names) - - values = dict(keys) - values['_blob'] = blob - - t.execute(query, values) - - def get_blob(self, **keys): - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - - column_names = list(keys.keys()) - - with self._sql.transaction() as t: - query = t.select_objects(self._blobtable, '_blob', *column_names) - blobs = [bytes(row['_blob']) for row in t.execute(query, keys)] - if len(blobs) == 0: - raise NoSuchObject(keys) - return blobs - - def remove_blob(self, **keys): - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - - column_names = list(keys.keys()) - with self._sql.transaction() as t: - query = t.remove_objects(self._blobtable, *column_names) - t.execute(query, keys) - - -class KeyCollision(Exception): - - def __init__(self, keys): - super().__init__('Cannot add object with same keys: %r' % keys) - - -class BlobKeyCollision(Exception): - - def __init__(self, subpath, keys): - super().__init__( - 'Cannot add blob with same keys: subpath=%s %r' % (subpath, keys)) - - -class UnknownKey(Exception): - - def __init__(self, key): - super().__init__('ObjectStore is not prepared for key %r' % key) - - -class WrongKeyType(Exception): - - def __init__(self, key, key_type): - super().__init__( - 'ObjectStore is not prepared for key %r of type %r, must be str' % - (key, key_type)) - - -class KeyValueError(Exception): - - def __init__(self, key, value): - super().__init__('Key %r value %r has the wrong type' % (key, value)) - - -class NoSuchObject(Exception): - - def __init__(self, keys): - super().__init__('No object/blob with keys {}'.format(keys)) - - -def flatten_object(obj): - # We sort only by the name, not the object in each pair in the - # list. Otherwise, if there are two fields with the same name but - # incompatible value types this will break. However, to guarantee - # that objects always result in the same flattened representation, - # we also compare the second field. For this, we convert the - # second value to a string with repr. This allows the second - # fields to be compared regardless of type. - - pairs = _flatten(obj) - unique_pairs = set(pairs) - sorted_pairs = sorted(unique_pairs, key=repr) - return list(sorted_pairs) - - -def _flatten(obj, obj_key=None): - if isinstance(obj, dict): - for key, value in obj.items(): - for x in _flatten(value, obj_key=key): - yield x - elif isinstance(obj, list): - for item in obj: - for x in _flatten(item, obj_key=obj_key): - yield x - else: - yield obj_key, obj diff --git a/qvarn/objstore_tests.py b/qvarn/objstore_tests.py deleted file mode 100644 index f8d631a..0000000 --- a/qvarn/objstore_tests.py +++ /dev/null @@ -1,244 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - -import qvarn - - -class ObjectStoreTests(unittest.TestCase): - - def setUp(self): - self.obj1 = { - 'name': 'this is my object', - } - self.obj2 = { - 'name': 'this is my other object', - } - self.blob1 = 'my first blob' - self.blob2 = 'my other blob' - - def create_store(self, **keys): - store = qvarn.MemoryObjectStore() - store.create_store(**keys) - return store - - def get_all_objects(self, store): - return [obj for _, obj in store.find_objects(qvarn.Yes())] - - def sorted_dicts(self, dicts): - return sorted(dicts, key=lambda d: sorted(d.items())) - - def test_is_initially_empty(self): - store = self.create_store(key=str) - self.assertEqual(self.get_all_objects(store), []) - - def test_refurses_nonstr_keys(self): - with self.assertRaises(qvarn.WrongKeyType): - self.create_store(key=int) - - def test_adds_object(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - self.assertEqual(self.get_all_objects(store), [self.obj1]) - - def test_adds_object_with_binary_data(self): - store = self.create_store(key=str) - self.obj1['data'] = bytes(range(0, 256)) - store.create_object(self.obj1, key='1st') - self.assertEqual(self.get_all_objects(store), [self.obj1]) - - def test_raises_error_for_surprising_keys(self): - store = self.create_store(key=str) - with self.assertRaises(qvarn.UnknownKey): - store.create_object(self.obj1, surprise='1st') - - def test_raises_error_adding_object_with_existing_keys(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - with self.assertRaises(qvarn.KeyCollision): - store.create_object(self.obj1, key='1st') - - def test_raises_error_adding_object_with_keys_of_wrong_type(self): - store = self.create_store(key=str) - with self.assertRaises(qvarn.KeyValueError): - store.create_object(self.obj1, key=1) - - def test_adds_objects_with_two_keys_with_one_key_the_same(self): - store = self.create_store(key1=str, key2=str) - store.create_object(self.obj1, key1='same', key2='1st') - store.create_object(self.obj2, key1='same', key2='2nd') - self.assertEqual(self.get_all_objects(store), [self.obj1, self.obj2]) - - def test_removes_only_object(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.remove_objects(key='1st') - self.assertEqual(self.get_all_objects(store), []) - - def test_gets_objects(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_object(self.obj2, key='2nd') - self.assertEqual(store.get_objects(key='1st'), [self.obj1]) - self.assertEqual(store.get_objects(key='2nd'), [self.obj2]) - - def test_gets_objects_using_only_one_key(self): - store = self.create_store(key1=str, key2=str) - store.create_object(self.obj1, key1='1st', key2='foo') - store.create_object(self.obj2, key1='2nd', key2='foo') - self.assertEqual(store.get_objects(key1='1st'), [self.obj1]) - self.assertEqual(store.get_objects(key1='2nd'), [self.obj2]) - self.assertEqual( - self.sorted_dicts(store.get_objects(key2='foo')), - self.sorted_dicts([self.obj1, self.obj2])) - - def test_removes_only_one_object(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_object(self.obj2, key='2nd') - store.remove_objects(key='1st') - self.assertEqual(self.get_all_objects(store), [self.obj2]) - - def test_finds_objects(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_object(self.obj2, key='2nd') - - cond = qvarn.Equal('name', self.obj1['name']) - objs = store.find_objects(cond) - self.assertEqual( - objs, - [({'key': '1st'}, self.obj1)] - ) - - def test_has_no_blob_initially(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - with self.assertRaises(qvarn.NoSuchObject): - store.get_blob(key='1st', subpath='blob') - - def test_add_blob_to_nonexistent_parent_fails(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - with self.assertRaises(qvarn.NoSuchObject): - store.create_blob(self.blob1, key='2nd', subpath='blob') - - def test_adds_blob(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_blob(self.blob1, key='1st', subpath='blob') - blob = store.get_blob(key='1st', subpath='blob') - self.assertEqual(blob, self.blob1) - - def test_add_blob_twice_fails(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_blob(self.blob1, key='1st', subpath='blob') - with self.assertRaises(qvarn.BlobKeyCollision): - store.create_blob(self.blob1, key='1st', subpath='blob') - - def test_removes_blob(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_blob(self.blob1, key='1st', subpath='blob') - store.remove_blob(key='1st', subpath='blob') - with self.assertRaises(qvarn.NoSuchObject): - store.get_blob(key='1st', subpath='blob') - - -class FlattenObjectsTests(unittest.TestCase): - - def test_flattens_simple_dict(self): - obj = { - 'foo': 'bar', - 'foobar': 42, - 'yo': True, - } - self.assertEqual( - qvarn.flatten_object(obj), - sorted([('foo', 'bar'), ('foobar', 42), ('yo', True)])) - - def test_flattens_deep_dict(self): - obj = { - 'foo': 'bar', - 'foos': [ - { - 'foo': 'bar2', - 'foos': [ - { - 'foo': 'bar3', - }, - ], - }, - ], - } - self.assertEqual( - qvarn.flatten_object(obj), - sorted([ - ('foo', 'bar'), - ('foo', 'bar2'), - ('foo', 'bar3'), - ])) - - -class FindObjectsTests(unittest.TestCase): - - def setUp(self): - pass - - def create_store(self, **keys): - store = qvarn.MemoryObjectStore() - store.create_store(**keys) - return store - - def test_finds_objects_matching_deeply_in_object(self): - obj1 = { - 'foo': 'foo-1', - 'bar': 'blah', - 'bars': [ - { - 'foo': 'bars.0', - 'bar': 'yo', - }, - { - 'foo': 'bars.1', - 'bar': 'bleurgh', - }, - ], - } - - obj2 = { - 'foo': 'foo-2', - 'bar': 'bother', - 'bars': [], - } - - keys1 = { - 'key': '1st', - } - - keys2 = { - 'key': '2nd', - } - - store = self.create_store(key=str) - store.create_object(obj1, **keys1) - store.create_object(obj2, **keys2) - - cond = qvarn.Equal('bar', 'yo') - objs = store.find_objects(cond) - self.assertEqual(objs, [(keys1, obj1)]) diff --git a/qvarn/resource_router.py b/qvarn/resource_router.py deleted file mode 100644 index c9df68a..0000000 --- a/qvarn/resource_router.py +++ /dev/null @@ -1,167 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class ResourceRouter(qvarn.Router): - - def __init__(self): - super().__init__() - self._coll = None - self._baseurl = None - self._notify = None - - def set_baseurl(self, baseurl): - self._baseurl = baseurl - - def set_collection(self, coll): - self._coll = coll - - def set_notifier(self, notify): - self._notify = notify - - def get_routes(self): - assert self._baseurl is not None - - rt = self._coll.get_type() - path = rt.get_path() - id_path = '{}/<id>'.format(path) - - return [ - { - 'method': 'POST', - 'path': path, - 'callback': self._create, - }, - { - 'method': 'PUT', - 'path': id_path, - 'callback': self._update, - }, - { - 'method': 'GET', - 'path': path, - 'callback': self._list, - }, - { - 'method': 'GET', - 'path': id_path, - 'callback': self._get, - }, - { - 'method': 'GET', - 'path': path + '/search/<search_criteria:path>', - 'callback': self._search, - }, - { - 'method': 'DELETE', - 'path': id_path, - 'callback': self._delete, - }, - ] - - def _create(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - if 'type' not in body: - body['type'] = self._coll.get_type_name() - - validator = qvarn.Validator() - try: - validator.validate_new_resource(body, self._coll.get_type()) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - result_body = self._coll.post(body) - qvarn.log.log( - 'debug', msg_text='POST a new resource, result', - body=result_body) - location = '{}{}/{}'.format( - self._baseurl, self._coll.get_type().get_path(), result_body['id']) - self._notify(result_body['id'], result_body['revision'], 'created') - return qvarn.created_response(result_body, location) - - def _update(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - if 'type' not in body: - body['type'] = self._coll.get_type_name() - - if 'id' not in body: - body['id'] = kwargs['id'] - - validator = qvarn.Validator() - try: - validator.validate_resource_update(body, self._coll.get_type()) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - obj_id = kwargs['id'] - # FIXME: the following test should be enabled once we - # no longer need test-api. - if False and body['id'] != obj_id: - raise qvarn.IdMismatch(body['id'], obj_id) - - try: - result_body = self._coll.put(body) - except qvarn.WrongRevision as e: - return qvarn.conflict_response(str(e)) - except qvarn.NoSuchResource as e: - # We intentionally say bad request, instead of not found. - # This is to be compatible with old Qvarn. This may get - # changed later. - return qvarn.bad_request_response(str(e)) - - self._notify(result_body['id'], result_body['revision'], 'updated') - return qvarn.ok_response(result_body) - - def _list(self, *args, **kwargs): - body = self._coll.list() - return qvarn.ok_response(body) - - def _get(self, *args, **kwargs): - try: - obj = self._coll.get(kwargs['id']) - except qvarn.NoSuchResource as e: - return qvarn.no_such_resource_response(str(e)) - return qvarn.ok_response(obj) - - def _search(self, *args, **kwargs): - path = kwargs['raw_uri_path'] - search_criteria = path.split('/search/', 1)[1] - try: - result = self._coll.search(search_criteria) - except qvarn.UnknownSearchField as e: - return qvarn.unknown_search_field_response(e) - except qvarn.NeedSortOperator: - return qvarn.need_sort_response() - except qvarn.SearchParserError as e: - return qvarn.search_parser_error_response(e) - body = { - 'resources': result, - } - return qvarn.ok_response(body) - - def _delete(self, *args, **kwargs): - obj_id = kwargs['id'] - self._coll.delete(obj_id) - self._notify(obj_id, None, 'deleted') - return qvarn.ok_response({}) diff --git a/qvarn/resource_type.py b/qvarn/resource_type.py deleted file mode 100644 index 40f8930..0000000 --- a/qvarn/resource_type.py +++ /dev/null @@ -1,150 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import os - -import yaml - - -class ResourceType: - - def __init__(self): - self._type = None - self._path = None - self._versions = [] - self._version = None - self._prototype = None - - def from_spec(self, spec): - self.set_type(spec['type']) - self.set_path(spec['path']) - self._versions = spec['versions'] - self.set_version_spec(self._versions[-1]) - - def as_dict(self): - return { - 'type': self.get_type(), - 'path': self.get_path(), - 'versions': [ - version_spec - for version_spec in self._versions - ] - } - - def set_type(self, type_name): - self._type = type_name - - def get_type(self): - return self._type - - def set_path(self, path): - self._path = path - - def get_path(self): - return self._path - - def set_version_spec(self, version_spec): - self._version = version_spec['version'] - self._prototype = version_spec['prototype'] - - def get_all_versions(self): - return [v['version'] for v in self._versions] - - def get_version(self, version): - for v in self._versions: - if v['version'] == version: - return v - raise KeyError('Version %r not found' % version) - - def get_latest_version(self): - return self._version - - def get_latest_prototype(self): - return self._prototype - - def get_subpaths(self): - v = self._versions[-1] - subpaths = v.get('subpaths', {}) - return { - subpath: subpaths[subpath]['prototype'] - for subpath in subpaths - } - - def get_subprototype(self, subpath): - v = self._versions[-1] - subpaths = v.get('subpaths', {}) - subproto = subpaths.get(subpath, {}) - return subproto.get('prototype') - - def get_files(self): - v = self._versions[-1] - return v.get('files', []) - - -def load_resource_types(dirname): # pragma: no cover - assert dirname is not None - resource_types = [] - basenames = [x for x in os.listdir(dirname) if x.endswith('.yaml')] - for basename in basenames: - pathname = os.path.join(dirname, basename) - with open(pathname) as f: - spec = yaml.safe_load(f) - rt = ResourceType() - rt.from_spec(spec) - resource_types.append(rt) - return resource_types - - -def add_missing_fields(proto, obj): - # Assume obj is validated. - - return _fill_in_dict(proto, obj) - - -def _fill_in_dict(proto, obj): - new = {} - defaults = { - str: '', - int: 0, - bool: False, - } - - for field in proto: - if type(proto[field]) in defaults: - if field not in obj: - new[field] = defaults[type(proto[field])] - elif isinstance(proto[field], list): - if field not in obj: - new[field] = [] - elif isinstance(proto[field][0], dict): - new[field] = [ - _fill_in_dict(proto[field][0], x) - for x in obj[field] - ] - elif type(proto[field][0]) in defaults: - new[field] = list(obj[field]) - else: # pragma: no cover - assert 0, 'field is {!r}'.format(field) - - if isinstance(obj, dict): # pragma: no cover - for field in obj: - if field not in new: - if isinstance(obj[field], list): - new[field] = list(obj[field]) - else: - new[field] = obj[field] - - return new diff --git a/qvarn/resource_type_tests.py b/qvarn/resource_type_tests.py deleted file mode 100644 index 36c775b..0000000 --- a/qvarn/resource_type_tests.py +++ /dev/null @@ -1,195 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - -import qvarn - - -class ResourceTypeTests(unittest.TestCase): - - def test_initially_has_no_type(self): - rt = qvarn.ResourceType() - self.assertEqual(rt.get_type(), None) - - def test_sets_type(self): - rt = qvarn.ResourceType() - rt.set_type('subject') - self.assertEqual(rt.get_type(), 'subject') - - def test_initially_has_no_path(self): - rt = qvarn.ResourceType() - self.assertEqual(rt.get_path(), None) - - def test_sets_path(self): - rt = qvarn.ResourceType() - rt.set_path('/subjects') - self.assertEqual(rt.get_path(), '/subjects') - - def test_initially_has_no_latest_version(self): - rt = qvarn.ResourceType() - self.assertEqual(rt.get_latest_version(), None) - - def test_load_resource_spec(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'foo': '', - }, - }, - { - 'version': 'v1', - 'prototype': { - 'foo': '', - 'bar': '', - 'version': 0, # test for bug - }, - 'subpaths': { - 'subfoo': { - 'prototype': { - 'subbar': '', - }, - }, - 'blob': { - 'prototype': { - 'blob': 'blob', - 'content-type': '', - }, - }, - }, - 'files': [ - 'blob', - ], - }, - ], - } - rt = qvarn.ResourceType() - rt.from_spec(spec) - self.assertEqual(rt.get_type(), spec['type']) - self.assertEqual(rt.get_path(), spec['path']) - self.assertEqual(rt.get_all_versions(), ['v0', 'v1']) - self.assertEqual(rt.get_version('v0'), spec['versions'][0]) - self.assertEqual(rt.get_version('v1'), spec['versions'][1]) - with self.assertRaises(KeyError): - rt.get_version('v999') - self.assertEqual( - rt.get_latest_version(), spec['versions'][-1]['version']) - self.assertEqual( - rt.get_latest_prototype(), spec['versions'][-1]['prototype']) - self.assertEqual(rt.as_dict(), spec) - subpaths = spec['versions'][-1]['subpaths'] - self.assertEqual( - rt.get_subpaths(), - { - 'subfoo': subpaths['subfoo']['prototype'], - 'blob': subpaths['blob']['prototype'], - }, - ) - self.assertEqual( - rt.get_subprototype('subfoo'), - spec['versions'][-1]['subpaths']['subfoo']['prototype'] - ) - self.assertEqual(rt.get_files(), ['blob']) - self.assertEqual( - rt.get_subprototype('blob'), - { - 'blob': 'blob', - 'content-type': '', - } - ) - - -class AddMissingFieldsTests(unittest.TestCase): - - def setUp(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v1', - 'prototype': { - 'foo': '', - 'bars': [ - { - 'foobar': '', - 'yo': '', - 'names': [''], - }, - ], - }, - }, - ], - } - self.rt = qvarn.ResourceType() - self.rt.from_spec(spec) - self.proto = self.rt.get_latest_prototype() - - def test_fills_in_toplevel_fields(self): - self.assertEqual( - qvarn.add_missing_fields(self.proto, {}), - { - 'foo': '', - 'bars': [], - } - ) - - def test_fills_in_list_dict_fields(self): - obj = { - 'bars': [ - { - }, - ], - } - self.assertEqual( - qvarn.add_missing_fields(self.proto, obj), - { - 'foo': '', - 'bars': [ - { - 'foobar': '', - 'yo': '', - 'names': [], - }, - ], - } - ) - - def test_fills_in_list_of_strings(self): - obj = { - 'bars': [ - { - 'names': ['James Bond'], - }, - ], - } - self.assertEqual( - qvarn.add_missing_fields(self.proto, obj), - { - 'foo': '', - 'bars': [ - { - 'foobar': '', - 'yo': '', - 'names': ['James Bond'], - }, - ], - } - ) diff --git a/qvarn/schema.py b/qvarn/schema.py deleted file mode 100644 index 7e533b8..0000000 --- a/qvarn/schema.py +++ /dev/null @@ -1,71 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -def schema(r): - stack = [] - push(stack, [], r) - result = [] - for x in generate(stack): - result.append(x) - return result - - -def push(stack, name, r): - stack.append((name, r)) - - -def pop(stack): - return stack.pop() - - -def generate(stack): - funcs = { - dict: dict_schema, - list: list_schema, - str: simple_schema, - int: simple_schema, - bool: simple_schema, - type(None): simple_schema, - } - - while stack: - name, r = pop(stack) - for x in funcs[type(r)](stack, name, r): - yield x - - -def dict_schema(stack, name, r): - for key in reversed(sorted(r.keys())): - push(stack, name + [key], r[key]) - return [] # must return an iterable - - -def list_schema(stack, name, r): - qvarn.log.log( - 'trace', msg_text='list_schema', stack=stack, name=name, len_r=len(r)) - if len(r) > 0: - yield name, list, type(r[0]) - if isinstance(r[0], dict): - push(stack, name, r[0]) - else: - yield name, list, None - - -def simple_schema(stack, name, r): - yield name, type(r) diff --git a/qvarn/schema_tests.py b/qvarn/schema_tests.py deleted file mode 100644 index 10461ad..0000000 --- a/qvarn/schema_tests.py +++ /dev/null @@ -1,90 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest -import sys - - -import qvarn - - -class SchemaTests(unittest.TestCase): - - def test_generates_schema_for_simple_resource_type(self): - resource_type = { - 'type': '', - } - self.assertEqual( - qvarn.schema(resource_type), - [ - (['type'], str), - ] - ) - - def test_generates_schema_with_simple_list(self): - resource_type = { - 'type': '', - 'foos': [''], - } - self.assertEqual( - qvarn.schema(resource_type), - [ - (['foos'], list, str), - (['type'], str), - ] - ) - - def test_generates_schema_with_dict_list(self): - resource_type = { - 'type': '', - 'foos': [ - { - 'yos': '', - 'bars': [''], - }, - ], - } - self.assertEqual( - qvarn.schema(resource_type), - [ - (['foos'], list, dict), - (['foos', 'bars'], list, str), - (['foos', 'yos'], str), - (['type'], str), - ] - ) - - def test_generates_schema_from_deep_resource_type(self): - N = sys.getrecursionlimit() + 1 - resource_type = { - 'foos': '', - } - for _ in range(N): - resource_type = { - 'foos': [resource_type], - } - self.assertTrue(isinstance(qvarn.schema(resource_type), list)) - - def test_generates_schema_from_dict_with_empty_list(self): - resource_type = { - 'foos': [], - } - self.assertEqual( - qvarn.schema(resource_type), - [ - (['foos'], list, None), - ] - ) diff --git a/qvarn/search_parser.py b/qvarn/search_parser.py deleted file mode 100644 index 7a90f64..0000000 --- a/qvarn/search_parser.py +++ /dev/null @@ -1,150 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import urllib - -import qvarn - - -class SearchParser: - - conditions = { - 'contains': (2, qvarn.Contains), - 'exact': (2, qvarn.Equal), - 'ge': (2, qvarn.GreaterOrEqual), - 'gt': (2, qvarn.GreaterThan), - 'le': (2, qvarn.LessOrEqual), - 'lt': (2, qvarn.LessThan), - 'ne': (2, qvarn.NotEqual), - 'startswith': (2, qvarn.Startswith), - 'show': (1, None), - 'show_all': (0, None), - 'sort': (1, None), - 'offset': (1, None), - 'limit': (1, None), - } - - def parse(self, path): - if not path: - raise SearchParserError('No condition given') - - sp = qvarn.SearchParameters() - - pairs = list(self._parse_simple(path)) - - for operator, args in pairs: - if operator == 'show_all': - sp.set_show_all() - elif operator == 'show': - for field in args: - sp.add_show_field(field) - elif operator == 'sort': - for field in args: - sp.add_sort_key(field) - elif operator == 'offset': - sp.set_offset(int(args[0])) - elif operator == 'limit': - sp.set_limit(int(args[0])) - else: - klass = self.conditions[operator][1] - cond = klass(*args) - sp.add_cond(cond) - - self._check_params(sp) - - return sp - - def _parse_simple(self, path): - # Yield operator, args pairs. - words = [self._unquote(w) for w in path.split('/')] - assert len(words) > 0 - while words: - operator, words = words[0], words[1:] - if operator not in self.conditions: - raise SearchParserError( - 'Unknown condition {}'.format(operator)) - - num_args = self.conditions[operator][0] - if num_args > len(words): - raise SearchParserError( - 'Not enough args for {}'.format(operator)) - - args, words = words[:num_args], words[num_args:] - yield operator, args - - def _unquote(self, word): - return urllib.parse.unquote(word) - - def _check_params(self, sp): - has_sort = sp.sort_keys != [] - has_offset = sp.offset is not None - has_limit = sp.limit is not None - if (has_limit or has_offset) and not has_sort: - raise NeedSortOperator() - - -class SearchParserError(Exception): - - def __init__(self, msg): - super().__init__(self, msg) - - -class NeedSortOperator(SearchParserError): - - def __init__(self): - super().__init__('/offset and /limit only valid with /sort') - - -class SearchParameters: - - def __init__(self): - self.sort_keys = [] - self.show_fields = [] - self.show_all = False - self.cond = None - self.offset = None - self.limit = None - - def set_offset(self, offset): - if self.offset is not None: - raise SearchParserError('/offset may only be used once') - self.offset = offset - - def set_limit(self, limit): - if self.limit is not None: - raise SearchParserError('/limit may only be used once') - self.limit = limit - - def add_sort_key(self, field_name): - self.sort_keys.append(field_name) - - def add_show_field(self, field_name): - if self.show_all: - raise SearchParserError('/show_all and /show conflict') - self.show_fields.append(field_name) - - def set_show_all(self): - if self.show_fields: - raise SearchParserError('/show_all and /show conflict') - self.show_all = True - - def add_cond(self, cond): - if self.cond is None: - self.cond = cond - elif isinstance(self.cond, qvarn.All): - self.cond.append_subcondition(cond) - else: - self.cond = qvarn.All(self.cond, cond) diff --git a/qvarn/search_parser_tests.py b/qvarn/search_parser_tests.py deleted file mode 100644 index 9f45325..0000000 --- a/qvarn/search_parser_tests.py +++ /dev/null @@ -1,207 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - - -import qvarn - - -class SearchParserTests(unittest.TestCase): - - def test_raises_error_path_is_empty(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('') - - def test_raises_error_if_condition_is_unknown(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('unknown/foo/bar') - - def test_returns_exact_condition(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, []) - self.assertEqual(sp.show_all, False) - self.assertTrue(isinstance(sp.cond, qvarn.Equal)) - self.assertEqual(sp.cond.name, 'foo') - self.assertEqual(sp.cond.pattern, 'bar') - - def test_handles_url_encoded_slash(self): - p = qvarn.SearchParser() - sp = p.parse('exact/operating_system/gnu%2Flinux') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, []) - self.assertEqual(sp.show_all, False) - self.assertTrue(isinstance(sp.cond, qvarn.Equal)) - self.assertEqual(sp.cond.name, 'operating_system') - self.assertEqual(sp.cond.pattern, 'gnu/linux') - - def test_raises_error_if_only_show_specified(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('show') - - def test_raises_error_if_both_show_and_show_all_specified(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('exact/foo/bar/show_all/show/foo') - - def test_raises_error_if_show_specified_but_without_operand(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('exact/foo/bar/show') - - def test_returns_show_if_specified(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/show/foo') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, ['foo']) - self.assertTrue(isinstance(sp.cond, qvarn.Equal)) - self.assertEqual(sp.cond.name, 'foo') - self.assertEqual(sp.cond.pattern, 'bar') - - def test_returns_show_all_if_specified(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/show_all') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_all, True) - self.assertTrue(isinstance(sp.cond, qvarn.Equal)) - self.assertEqual(sp.cond.name, 'foo') - self.assertEqual(sp.cond.pattern, 'bar') - - def test_returns_all_condition(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/exact/foobar/yo') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, []) - self.assertTrue(isinstance(sp.cond, qvarn.All)) - self.assertEqual(len(sp.cond.conds), 2) - first, second = sp.cond.conds - self.assertTrue(isinstance(first, qvarn.Equal)) - self.assertEqual(first.name, 'foo') - self.assertEqual(first.pattern, 'bar') - self.assertTrue(isinstance(second, qvarn.Equal)) - self.assertEqual(second.name, 'foobar') - self.assertEqual(second.pattern, 'yo') - - def test_returns_sort_keys(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/exact/foobar/yo/sort/a/sort/b') - self.assertEqual(sp.sort_keys, ['a', 'b']) - - def test_returns_sort_keys_with_show_all(self): - p = qvarn.SearchParser() - sp = p.parse('show_all/exact/foo/bar/exact/foobar/yo/sort/a/sort/b') - self.assertEqual(sp.sort_keys, ['a', 'b']) - - def test_sets_offset_and_limit(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/sort/a/offset/42/limit/128') - self.assertEqual(sp.offset, 42) - self.assertEqual(sp.limit, 128) - - def test_raises_error_for_offset_without_sort(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.NeedSortOperator): - p.parse('offset/1') - - def test_raises_error_for_limit_without_sort(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.NeedSortOperator): - p.parse('limit/1') - - def test_accepts_limit_without_offset(self): - p = qvarn.SearchParser() - sp = p.parse('sort/x/limit/1') - self.assertEqual(sp.offset, None) - self.assertEqual(sp.limit, 1) - - -class SearchParametersTest(unittest.TestCase): - - def test_has_correct_initial_state(self): - sp = qvarn.SearchParameters() - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, []) - self.assertEqual(sp.show_all, False) - self.assertEqual(sp.cond, None) - - def test_sets_offset(self): - sp = qvarn.SearchParameters() - sp.set_offset(42) - self.assertEqual(sp.offset, 42) - - def test_raises_error_if_setting_offset_a_second_time(self): - sp = qvarn.SearchParameters() - sp.set_offset(42) - with self.assertRaises(qvarn.SearchParserError): - sp.set_offset(42) - - def test_sets_limit(self): - sp = qvarn.SearchParameters() - sp.set_limit(42) - self.assertEqual(sp.limit, 42) - - def test_raises_error_if_setting_limit_a_second_time(self): - sp = qvarn.SearchParameters() - sp.set_limit(42) - with self.assertRaises(qvarn.SearchParserError): - sp.set_limit(42) - - def test_adds_sort_key(self): - sp = qvarn.SearchParameters() - sp.add_sort_key('foo') - self.assertEqual(sp.sort_keys, ['foo']) - - def test_adds_show_field(self): - sp = qvarn.SearchParameters() - sp.add_show_field('foo') - self.assertEqual(sp.show_fields, ['foo']) - - def test_set_show_all(self): - sp = qvarn.SearchParameters() - sp.set_show_all() - self.assertEqual(sp.show_all, True) - - def test_show_when_show_all_is_set_raises_error(self): - sp = qvarn.SearchParameters() - sp.set_show_all() - with self.assertRaises(qvarn.SearchParserError): - sp.add_show_field('foo') - - def test_setting_show_all_when_fields_are_set_raises_error(self): - sp = qvarn.SearchParameters() - sp.add_show_field('foo') - with self.assertRaises(qvarn.SearchParserError): - sp.set_show_all() - - def test_adds_cond(self): - cond = qvarn.Yes() - sp = qvarn.SearchParameters() - self.assertEqual(sp.cond, None) - sp.add_cond(cond) - self.assertEqual(sp.cond, cond) - sp.add_cond(cond) - self.assertTrue(isinstance(sp.cond, qvarn.All)) - self.assertEqual(len(sp.cond.conds), 2) - self.assertEqual(sp.cond.conds, [cond, cond]) - sp.add_cond(cond) - self.assertTrue(isinstance(sp.cond, qvarn.All)) - self.assertEqual(len(sp.cond.conds), 3) - self.assertEqual(sp.cond.conds, [cond, cond, cond]) diff --git a/qvarn/sql.py b/qvarn/sql.py deleted file mode 100644 index 604272b..0000000 --- a/qvarn/sql.py +++ /dev/null @@ -1,339 +0,0 @@ -# sql.py - SQL dialect adaptation -# -# Copyright (C) 2017 Lars Wirzenius - - -'''Communicate with a PostgreSQL server.''' - - -import psycopg2 -import psycopg2.pool -import psycopg2.extras -import psycopg2.extensions - -import slog - -import qvarn - - -class PostgresAdapter: - - def __init__(self): - self._pool = None - - def connect(self, **kwargs): - self._pool = psycopg2.pool.ThreadedConnectionPool( - minconn=kwargs['min_conn'], - maxconn=kwargs['max_conn'], - database=kwargs['database'], - user=kwargs['user'], - password=kwargs['password'], - host=kwargs['host'], - port=kwargs['port'], - ) - - def transaction(self): - return Transaction(self) - - def get_conn(self): - return self._pool.getconn() - - def put_conn(self, conn): - self._pool.putconn(conn) - - -class Transaction: - - def __init__(self, sql): - self._sql = sql - self._conn = None - - def __enter__(self): - self._conn = self._sql.get_conn() - return self - - def __exit__(self, exc_type, exc_val, exc_tb): - try: - if exc_type is None: - self._conn.commit() - else: # pragma: no cover - self._conn.rollback() - except BaseException: # pragma: no cover - self._sql.put_conn(self._conn) - self._conn = None - raise - self._sql.put_conn(self._conn) - self._conn = None - - def execute(self, query, values): - qvarn.log.log('trace', msg_text='executing SQL query', query=query) - c = self._conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) - c.execute(query, values) - return c - - def get_rows(self, cursor): - for row in cursor: - yield dict(row) - - def create_table(self, table_name, **keys): - columns = ', '.join( - '{} {}'.format(self._q(key_name), self._sqltype(col_type)) - for key_name, col_type in keys.items() - ) - return 'CREATE TABLE IF NOT EXISTS {} ({})'.format( - self._q(table_name), - columns) - - def _sqltype(self, col_type): - types = [ - (str, 'TEXT'), - (int, 'BIGINT'), - (bool, 'BOOL'), - (dict, 'JSONB'), - (bytes, 'BYTEA'), - ] - for t, n in types: - if col_type == t: - return n - assert False - - def insert_object(self, table_name, *keys): - columns = [self._q(k) for k in keys] - placeholders = [self._placeholder(k) for k in keys] - return 'INSERT INTO {} ({}) VALUES ({})'.format( - self._q(table_name), - ', '.join(columns), - ', '.join(placeholders), - ) - - def remove_objects(self, table_name, *keys): - conditions = [ - '{} = {}'.format(self._q(key), self._placeholder(key)) - for key in keys - ] - return 'DELETE FROM {} WHERE {}'.format( - self._q(table_name), - ' AND '.join(conditions), - ) - - def select_objects(self, table_name, col_name, *keys): - conditions = [ - '{} = {}'.format(self._q(key), self._placeholder(key)) - for key in keys - ] - query = 'SELECT {} FROM {}'.format( - self._q(col_name), - self._q(table_name), - ) - if conditions: - query += ' WHERE {}'.format(' AND '.join(conditions)) - qvarn.log.log( - 'debug', msg_text='PostgresAdapater.select_objects', - query=query, keys=keys, conditions=conditions) - return query - - def select_objects_on_cond(self, table_name, cond, *keys): - query, values = qvarn.sql_select(_counter, cond) - return query, values - - def _condition(self, cond): - return cond.as_sql() - - def _q(self, name): - return quote(name) - - def _placeholder(self, name): - return placeholder(name) - - -def quote(name): - ascii_lower = 'abcdefghijklmnopqrstuvwxyz' - ascii_digits = '0123456789' - ascii_chars = ascii_lower + ascii_lower.upper() + ascii_digits - ok = ascii_chars + '_' - assert name.strip(ok) == '', 'must have only allowed chars: %r' % name - return '_'.join(name.split('-')) - - -def placeholder(name): - return '%({})s'.format(quote(name)) - - -_counter = slog.Counter() - - -def get_unique_name(base, counter=None): - if counter is None: - counter = _counter - return '{}{}'.format(base, counter.increment()) - - -class Condition: - - def get_subconditions(self): - return [] - - def matches(self, obj): # pragma: no cover - raise NotImplementedError() - - def as_sql(self): # pragma: no cover - raise NotImplementedError() - - -class All(Condition): - - def __init__(self, *conds): - self.conds = list(conds) - - def append_subcondition(self, cond): - self.conds.append(cond) - - def get_subconditions(self): - return self.conds - - def matches(self, obj): - for cond in self.conds: - if not cond.matches(obj): - return False - return True - - def as_sql(self): # pragma: no cover - pairs = [cond.as_sql() for cond in self.conds] - conds = ' AND '.join(query for query, _ in pairs) - values = {} - for _, value in pairs: - values.update(value) - return '( {} )'.format(conds), values - - -class Cmp(Condition): - - def __init__(self, name, pattern): - self.name = name - self.pattern = pattern - - def cmp_py(self, actual): - raise NotImplementedError() - - def cmp_sql(self, pattern_name): - return "lower(_field->>'value') {} lower(%({})s)".format( - self.get_operator(), pattern_name) - - def get_operator(self): - raise NotImplementedError() - - def matches(self, obj): - for key, actual in qvarn.flatten_object(obj): - if key == self.name and self.cmp_py(actual): - return True - return False - - def as_sql(self): # pragma: no cover - name_name = get_unique_name('name') - pattern_name = get_unique_name('pattern') - values = { - name_name: self.name, - pattern_name: self.pattern, - } - query = ("_field ->> 'name' = %%(%s)s AND " - "_field ->> 'value' %s") % ( - name_name, self.cmp_sql(pattern_name)) - return query, values - - -class Equal(Cmp): - - def cmp_py(self, actual): - return self.pattern.lower() == actual.lower() - - def get_operator(self): - return '=' - - -class ResourceTypeIs(Equal): - - def __init__(self, type_name): - super().__init__('type', type_name) - - def matches(self, obj): - return obj.get('type') == self.pattern - - -class NotEqual(Cmp): - - def cmp_py(self, actual): - return self.pattern.lower() != actual.lower() - - def get_operator(self): - return '!=' - - -class GreaterThan(Cmp): - - def cmp_py(self, actual): - return actual.lower() > self.pattern.lower() - - def get_operator(self): - return '>' - - -class GreaterOrEqual(Cmp): - - def cmp_py(self, actual): - return actual.lower() >= self.pattern.lower() - - def get_operator(self): - return '>=' - - -class LessThan(Cmp): - - def cmp_py(self, actual): - return actual.lower() < self.pattern.lower() - - def get_operator(self): - return '<' - - -class LessOrEqual(Cmp): - - def cmp_py(self, actual): - return actual.lower() <= self.pattern.lower() - - def get_operator(self): - return '<=' - - -class Contains(Cmp): - - def cmp_py(self, actual): - return self.pattern.lower() in actual.lower() - - def cmp_sql(self, pattern_name): - t = "lower(_field->>'value') LIKE '%%' || lower(%({})s) || '%%'" - return t.format(pattern_name) - - def get_operator(self): - pass - - -class Startswith(Cmp): - - def cmp_py(self, actual): - return actual.lower().startswith(self.pattern.lower()) - - def cmp_sql(self, pattern_name): - t = "lower(_field->>'value') LIKE lower(%({})s) || '%%'" - return t.format(pattern_name) - - def get_operator(self): - pass - - -class Yes(Condition): - - def matches(self, obj): - return True - - def as_sql(self): # pragma: no cover - return 'TRUE', {} diff --git a/qvarn/sql_select.py b/qvarn/sql_select.py deleted file mode 100644 index e54a5b9..0000000 --- a/qvarn/sql_select.py +++ /dev/null @@ -1,85 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -def sql_select(counter, cond): - conds = list(flatten(cond)) - if len(conds) == 1: - query, params = _select_on_simple_cond(counter, cond) - else: - query, params = _select_on_multiple_conds(counter, conds) - return query, params - - -def _select_on_simple_cond(counter, cond): - name = qvarn.get_unique_name('name', counter=counter) - value = qvarn.get_unique_name('value', counter=counter) - params = { - name: cond.name, - value: cond.pattern, - } - template = ' '.join(''' - SELECT _objects.obj_id, _objects.subpath, _objects._obj FROM _objects, - ( - SELECT obj_id FROM _aux WHERE - _field->>'name' = %({name})s - AND {valuecmp} - ) AS _temp - WHERE _temp.obj_id = _objects.obj_id - '''.split()) - query = template.format(name=name, valuecmp=cond.cmp_sql(value)) - return query, params - - -def _select_on_multiple_conds(counter, conds): - params = { - 'count': len(conds), - } - - part_template = "(_field->>'name' = %({name})s AND {valuecmp})" - parts = [] - for subcond in conds: - name = qvarn.get_unique_name('name', counter=counter) - value = qvarn.get_unique_name('value', counter=counter) - params[name] = subcond.name - params[value] = subcond.pattern - part = part_template.format(name=name, valuecmp=subcond.cmp_sql(value)) - parts.append(part) - - template = ' '.join(''' - SELECT _objects.obj_id, _objects.subpath, _objects._obj - FROM _objects, ( - SELECT obj_id, count(obj_id) AS _hits FROM _aux WHERE - {} - GROUP BY obj_id - ) AS _temp WHERE _hits >= %(count)s AND _temp.obj_id = _objects.obj_id - '''.split()) - - query = template.format(' OR '.join(parts)) - - return query, params - - -def flatten(cond): - subs = cond.get_subconditions() - if subs: - for sub in subs: - for c in flatten(sub): - yield c - else: - yield cond diff --git a/qvarn/sql_select_tests.py b/qvarn/sql_select_tests.py deleted file mode 100644 index 6cf3e56..0000000 --- a/qvarn/sql_select_tests.py +++ /dev/null @@ -1,96 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - -import slog - -import qvarn - - -class SqlSelectTests(unittest.TestCase): - - def test_returns_simple_query_for_simple_equal(self): - cond = qvarn.Equal('foo', 'bar') - counter = slog.Counter() - sql, values = qvarn.sql_select(counter, cond) - self.assertEqual( - sql, - ("SELECT _objects.obj_id, _objects.subpath, _objects._obj " - "FROM _objects, " - "( SELECT obj_id FROM _aux WHERE " - "_field->>'name' = %(name1)s AND " - "lower(_field->>'value') = lower(%(value2)s) ) AS _temp " - "WHERE _temp.obj_id = _objects.obj_id") - ) - self.assertEqual( - values, - { - 'name1': 'foo', - 'value2': 'bar', - } - ) - - def test_returns_simple_query_for_simple_not_equal(self): - cond = qvarn.NotEqual('foo', 'bar') - counter = slog.Counter() - sql, values = qvarn.sql_select(counter, cond) - self.assertEqual( - sql, - ("SELECT _objects.obj_id, _objects.subpath, _objects._obj " - "FROM _objects, " - "( SELECT obj_id FROM _aux WHERE " - "_field->>'name' = %(name1)s AND " - "lower(_field->>'value') != lower(%(value2)s) ) AS _temp " - "WHERE _temp.obj_id = _objects.obj_id") - ) - self.assertEqual( - values, - { - 'name1': 'foo', - 'value2': 'bar', - } - ) - - def test_returns_query_for_anded_conditions(self): - cond1 = qvarn.Equal('foo1', 'bar1') - cond2 = qvarn.NotEqual('foo2', 'bar2') - cond = qvarn.All(cond1, cond2) - counter = slog.Counter() - sql, values = qvarn.sql_select(counter, cond) - self.maxDiff = None - self.assertEqual( - sql, - ("SELECT _objects.obj_id, _objects.subpath, _objects._obj " - "FROM _objects, ( " - "SELECT obj_id, count(obj_id) AS _hits FROM _aux WHERE " - "(_field->>'name' = %(name1)s AND " - "lower(_field->>'value') = lower(%(value2)s)) OR " - "(_field->>'name' = %(name3)s AND " - "lower(_field->>'value') != lower(%(value4)s)) " - "GROUP BY obj_id ) AS _temp WHERE _hits >= %(count)s AND " - "_temp.obj_id = _objects.obj_id") - ) - self.assertEqual( - values, - { - 'name1': 'foo1', - 'value2': 'bar1', - 'name3': 'foo2', - 'value4': 'bar2', - 'count': 2, - } - ) diff --git a/qvarn/subresource_router.py b/qvarn/subresource_router.py deleted file mode 100644 index 343ee4a..0000000 --- a/qvarn/subresource_router.py +++ /dev/null @@ -1,82 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class SubresourceRouter(qvarn.Router): - - def __init__(self): - super().__init__() - self._parent_coll = None - self._subpath = None - - def set_subpath(self, subpath): - self._subpath = subpath - - def set_parent_collection(self, parent_coll): - self._parent_coll = parent_coll - - def get_routes(self): - rt = self._parent_coll.get_type() - path = '{}/<id>/{}'.format(rt.get_path(), self._subpath) - return [ - { - 'method': 'GET', - 'path': path, - 'callback': self._get_subresource, - }, - { - 'method': 'PUT', - 'path': path, - 'callback': self._put_subresource, - }, - ] - - def _get_subresource(self, *args, **kwargs): - obj_id = kwargs['id'] - try: - obj = self._parent_coll.get_subresource(obj_id, self._subpath) - except qvarn.NoSuchResource as e: - return qvarn.no_such_resource_response(str(e)) - return qvarn.ok_response(obj) - - def _put_subresource(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - obj_id = kwargs['id'] - if 'revision' not in body: - return qvarn.bad_request_response('must have revision') - revision = body.pop('revision') - - rt = self._parent_coll.get_type() - validator = qvarn.Validator() - try: - validator.validate_subresource(self._subpath, rt, body) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - try: - result_body = self._parent_coll.put_subresource( - body, subpath=self._subpath, obj_id=obj_id, revision=revision) - except qvarn.WrongRevision as e: - return qvarn.conflict_response(str(e)) - except qvarn.NoSuchResource as e: - return qvarn.no_such_resource_response(str(e)) - - return qvarn.ok_response(result_body) diff --git a/qvarn/validator.py b/qvarn/validator.py deleted file mode 100644 index 725d947..0000000 --- a/qvarn/validator.py +++ /dev/null @@ -1,129 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class Validator: - - def validate_against_prototype( - self, resource_type_name, resource, prototype): - actual_schema = qvarn.schema(resource) - wanted_schema = qvarn.schema(prototype) - - allowed_names = [x[0] for x in wanted_schema] - for actual in actual_schema: - if actual[0] not in allowed_names: - dotted = '.'.join(actual[0]) - raise UnknownField(resource_type_name, dotted) - - def _validate(self, resource, resource_type): - if not isinstance(resource, dict): - raise NotADict(resource) - if 'type' not in resource: - raise NoType() - if resource['type'] != resource_type.get_type(): - raise WrongType(resource['type'], resource_type.get_type()) - - prototype = resource_type.get_latest_prototype() - self.validate_against_prototype(resource['type'], resource, prototype) - - def validate_new_resource(self, resource, resource_type): - self._validate(resource, resource_type) - if 'id' in resource: - raise HasId() - if 'revision' in resource: - raise HasRevision() - - def validate_resource_update(self, resource, resource_type): - self._validate(resource, resource_type) - if 'id' not in resource: - raise NoId() - if 'revision' not in resource: - raise NoRevision() - - def validate_subresource(self, subpath, resource_type, sub): - qvarn.log.log( - 'debug', msg_text='validating subresource', subpath=subpath, - sub=sub) - subproto = resource_type.get_subprototype(subpath) - if subproto is None: - raise UnknownSubpath(resource_type.get_type(), subpath) - self.validate_against_prototype('FIXME', sub, subproto) - - -class ValidationError(Exception): - - pass - - -class NotADict(ValidationError): # pragma: no cover - - def __init__(self, resource): - super().__init__('Was expecting a dict, got %r' % type(resource)) - - -class NoType(ValidationError): - - def __init__(self): - super().__init__("Resources MUST have a type field") - - -class WrongType(ValidationError): - - def __init__(self, actual, expected): - super().__init__( - 'Resource has type %s, but %s was expected' % (actual, expected)) - - -class NoId(ValidationError): - - def __init__(self): - super().__init__("PUTted resources MUST have an id set") - - -class HasId(ValidationError): - - def __init__(self): - super().__init__("POSTed resources MUST NOT have an id set") - - -class NoRevision(ValidationError): - - def __init__(self): - super().__init__("PUTted resources MUST have a revision set") - - -class HasRevision(ValidationError): - - def __init__(self): - super().__init__("POSTed resources MUST NOT have a revision set") - - -class UnknownField(ValidationError): # pragma: no cover - - def __init__(self, type_name, name): - super().__init__( - 'Resource type {} has unknown field {}'.format( - type_name, name)) - - -class UnknownSubpath(ValidationError): # pragma: no cover - - def __init__(self, type_name, subpath): - super().__init__( - 'Resource type {} has not sub-resource {}'.format( - type_name, subpath)) diff --git a/qvarn/validator_tests.py b/qvarn/validator_tests.py deleted file mode 100644 index 8322f89..0000000 --- a/qvarn/validator_tests.py +++ /dev/null @@ -1,131 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - - -import qvarn - - -class ValidatorTests(unittest.TestCase): - - def setUp(self): - self.spec = { - 'type': 'foo', - 'path': '/foos', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'type': '', - 'id': '', - 'revision': '', - 'foo': '', - }, - 'subpaths': { - 'sub': { - 'prototype': { - 'subfield': '', - }, - }, - }, - }, - ] - } - - self.resource_type = qvarn.ResourceType() - self.resource_type.from_spec(self.spec) - - self.validator = qvarn.Validator() - - self.resource = { - 'type': 'foo', - 'id': 'resource-1', - 'revision': 'revision-1', - 'foo': 'this is foo', - } - - self.subresource = { - 'subfield': 'bananarama', - } - - def test_accepts_valid_resource(self): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_rejects_resource_that_is_not_a_dict(self): - with self.assertRaises(qvarn.NotADict): - self.validator.validate_new_resource(None, self.resource_type) - - def test_rejects_resource_without_type(self): - del self.resource['type'] - with self.assertRaises(qvarn.NoType): - self.validator.validate_new_resource( - self.resource, self.resource_type) - - def test_rejects_new_resource_with_different_type(self): - self.resource['type'] = 'wrong' - del self.resource['id'] - del self.resource['revision'] - with self.assertRaises(qvarn.WrongType): - self.validator.validate_new_resource( - self.resource, self.resource_type) - - def test_rejects_resource_update_with_different_type(self): - self.resource['type'] = 'wrong' - with self.assertRaises(qvarn.WrongType): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_rejects_resource_without_id(self): - del self.resource['id'] - with self.assertRaises(qvarn.NoId): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_rejects_resource_with_id(self): - with self.assertRaises(qvarn.HasId): - self.validator.validate_new_resource( - self.resource, self.resource_type) - - def test_rejects_resource_with_revision(self): - del self.resource['id'] - with self.assertRaises(qvarn.HasRevision): - self.validator.validate_new_resource( - self.resource, self.resource_type) - - def test_rejects_resource_without_revision(self): - del self.resource['revision'] - with self.assertRaises(qvarn.NoRevision): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_rejects_resource_with_unknown_field(self): - self.resource['unkown'] = '' - with self.assertRaises(qvarn.UnknownField): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_accepts_valid_subresource(self): - self.assertEqual( - self.validator.validate_subresource( - 'sub', self.resource_type, self.subresource), - None) - - def test_rejects_unknown_subresouce_path(self): - with self.assertRaises(qvarn.UnknownSubpath): - self.validator.validate_subresource( - 'unknown', self.resource_type, None) diff --git a/qvarn/version.py b/qvarn/version.py deleted file mode 100644 index bf504d8..0000000 --- a/qvarn/version.py +++ /dev/null @@ -1,2 +0,0 @@ -__version__ = "0.86+git" -__version_info__ = (0, 86, '+git') diff --git a/randport b/randport deleted file mode 100755 index d523401..0000000 --- a/randport +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/python3 -# -# Copyright (C) 2017 Lars Wirzenius -# -# Pick a random port that is free to be listened on. For testing. - - -import errno -import random -import socket -import sys - - -MAX = 1000 -for i in range(MAX): - port = random.randint(1025, 2**15-1) - s = socket.socket() - try: - s.bind(('0.0.0.0', port)) - except OSError as e: - if e.errno == errno.EADDRINUSE: - continue - raise - break -else: - sys.stderr.write("Can't find a free port\n") - sys.exit(1) -sys.stdout.write('{}\n'.format(port)) diff --git a/resource_type/bolagsfakta_suppliers.yaml b/resource_type/bolagsfakta_suppliers.yaml deleted file mode 100644 index 8dda7b2..0000000 --- a/resource_type/bolagsfakta_suppliers.yaml +++ /dev/null @@ -1,17 +0,0 @@ -path: /bolagsfakta_suppliers -type: bolagsfakta_supplier -versions: -- prototype: - bolagsfakta_status: '' - contract_end_date: '' - contract_start_date: '' - id: '' - materialized_path: [''] - parent_org_id: '' - parent_supplier_id: '' - project_resource_id: '' - revision: '' - supplier_org_id: '' - supplier_type: '' - type: '' - version: v0 diff --git a/resource_type/cards.yaml b/resource_type/cards.yaml deleted file mode 100644 index 6fef7b7..0000000 --- a/resource_type/cards.yaml +++ /dev/null @@ -1,56 +0,0 @@ -path: /cards -type: card -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- files: [holder_photo, issuer_logo] - prototype: - card_ids: - - {card_id: '', card_id_type: ''} - card_status_history: - - {card_status: '', modification_description: '', modification_reason: '', modified_by: '', - modified_timestamp: ''} - card_type: '' - created_date: '' - current_status: '' - expiration_date: '' - holder_gov_ids: - - {country: '', gov_id: '', id_type: ''} - holder_names: - - full_name: '' - given_names: [''] - sort_key: '' - surnames: [''] - titles: [''] - holder_nationalities: [''] - id: '' - id06_supplier_full_name: '' - id06_taxation_country: '' - id06_virtual_bankid_required: false - id06_virtual_devices: [''] - id06_virtual_enabled: false - id06_virtual_history: - - {date: '', enabled: false, full_name: ''} - id06_virtual_valid_from: '' - id06_virtual_valid_until: '' - issued_date: '' - issuer_gov_org_ids: - - {country: '', gov_org_id: '', org_id_type: ''} - issuer_name: '' - org: '' - person: '' - revision: '' - type: '' - valid_from_date: '' - valid_until_date: '' - subpaths: - holder_photo: - prototype: {body: blob, content_type: ''} - issuer_logo: - prototype: {body: blob, content_type: ''} - sync: - prototype: - sync_revision: '' - sync_sources: - - {sync_id: '', sync_source: ''} - version: v1 diff --git a/resource_type/competence_types.yaml b/resource_type/competence_types.yaml deleted file mode 100644 index 5bc003c..0000000 --- a/resource_type/competence_types.yaml +++ /dev/null @@ -1,107 +0,0 @@ -path: /competence_types -type: competence_type -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: &id001 - competence_type_id: '' - descriptions: - - {description: '', locale: ''} - id: '' - names: - - {locale: '', name: ''} - revision: '' - type: '' - version: v1 -- files: [card_front, card_back, registry_logo] - prototype: *id001 - subpaths: - card_back: - prototype: &id002 {body: blob, content_type: ''} - card_front: - prototype: &id003 {body: blob, content_type: ''} - registry_logo: - prototype: &id004 {body: blob, content_type: ''} - version: v2 -- files: [card_front, card_back, registry_logo] - prototype: - competence_type_id: '' - customer_org_ids: [''] - descriptions: - - {description: '', locale: ''} - id: '' - names: - - {locale: '', name: ''} - revision: '' - type: '' - subpaths: - card_back: - prototype: *id002 - card_front: - prototype: *id003 - registry_logo: - prototype: *id004 - version: v3 -- files: [card_front, card_back, registry_logo] - prototype: - competence_type_id: '' - customer_org_ids: [''] - descriptions: - - {description: '', locale: ''} - id: '' - master_register_name: '' - master_register_org_id: '' - names: - - {locale: '', name: ''} - revision: '' - type: '' - subpaths: - card_back: - prototype: *id002 - card_front: - prototype: *id003 - registry_logo: - prototype: *id004 - version: v4 -- files: [card_front, card_back, registry_logo] - prototype: - competence_type_id: '' - customer_org_ids: [''] - descriptions: - - {description: '', locale: ''} - id: '' - is_verified: false - names: - - {locale: '', name: ''} - revision: '' - type: '' - subpaths: - card_back: - prototype: *id002 - card_front: - prototype: *id003 - registry_logo: - prototype: *id004 - version: v5 -- files: [card_front, card_back, registry_logo] - prototype: - attribute_validation: - - {attribute: '', regex: '', required: false} - competence_type_id: '' - customer_org_ids: [''] - descriptions: - - {description: '', locale: ''} - id: '' - is_verified: false - names: - - {locale: '', name: ''} - revision: '' - type: '' - subpaths: - card_back: - prototype: *id002 - card_front: - prototype: *id003 - registry_logo: - prototype: *id004 - version: v6 diff --git a/resource_type/competences.yaml b/resource_type/competences.yaml deleted file mode 100644 index 9f6a301..0000000 --- a/resource_type/competences.yaml +++ /dev/null @@ -1,47 +0,0 @@ -path: /competences -type: competence -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: - competence_card_holder_names: - - full_name: '' - given_names: [''] - sort_key: '' - surnames: [''] - competence_card_id: '' - competence_type_id: '' - employee_person_id: '' - employer_org_id: '' - id: '' - revision: '' - type: '' - valid_from_date: '' - valid_until_date: '' - subpaths: - sync: - prototype: &id001 {sync_id: '', sync_revision: ''} - version: v1 -- prototype: - competence_card_holder_names: - - full_name: '' - given_names: [''] - sort_key: '' - surnames: [''] - competence_card_id: '' - competence_registry_id: '' - competence_trainer: '' - competence_type_id: '' - employee_person_id: '' - employer_org_id: '' - granted_by_org_id: '' - id: '' - revision: '' - type: '' - valid_from_date: '' - valid_until_date: '' - validation_status: '' - subpaths: - sync: - prototype: *id001 - version: v2 diff --git a/resource_type/contracts.yaml b/resource_type/contracts.yaml deleted file mode 100644 index 6cf8bc1..0000000 --- a/resource_type/contracts.yaml +++ /dev/null @@ -1,416 +0,0 @@ -path: /contracts -type: contract -versions: - -- version: v0 - prototype: - id: '' - revision: '' - type: '' - -- version: v1 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v2 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v3 - prototype: - contract_parties: - - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v4 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v5 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - global_permissions: - - permission_name: '' - permissions: - - permission_name: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v6 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - global_permissions: - - permission_name: '' - permissions: - - permission_name: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - terms_of_service: - - acceptance_time: '' - accepter_person_id: '' - terms_of_service_version: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v7 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - global_permissions: - - permission_name: '' - permissions: - - permission_name: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - terms_of_service: - - acceptance_time: '' - accepter_person_id: '' - terms_of_service_language: '' - terms_of_service_version: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document diff --git a/resource_type/data_cache.yaml b/resource_type/data_cache.yaml deleted file mode 100644 index d1574b7..0000000 --- a/resource_type/data_cache.yaml +++ /dev/null @@ -1,8 +0,0 @@ -path: /data_cache -type: data_cache -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: {cache_name: '', expires_at: '', id: '', key: '', revision: '', type: '', - value: ''} - version: v1 diff --git a/resource_type/events.yaml b/resource_type/events.yaml deleted file mode 100644 index 589ac85..0000000 --- a/resource_type/events.yaml +++ /dev/null @@ -1,14 +0,0 @@ -path: /events -type: event -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: {card: '', card_event_type: '', event_type: '', generated_timestamp: '', - id: '', org: '', person: '', project: '', revision: '', type: ''} - subpaths: - sync: - prototype: - sync_revision: '' - sync_sources: - - {sync_id: '', sync_source: ''} - version: v1 diff --git a/resource_type/files.yaml b/resource_type/files.yaml deleted file mode 100644 index ef74217..0000000 --- a/resource_type/files.yaml +++ /dev/null @@ -1,16 +0,0 @@ -path: /files -type: file -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- files: [file] - prototype: - filename: '' - id: '' - revision: '' - type: '' - version: 0 - subpaths: - file: - prototype: {body: blob, content_type: ''} - version: v1 diff --git a/resource_type/jobs.yaml b/resource_type/jobs.yaml deleted file mode 100644 index a4222b0..0000000 --- a/resource_type/jobs.yaml +++ /dev/null @@ -1,20 +0,0 @@ -path: /jobs -type: job -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: - done_at: '' - id: '' - job_type: '' - org_id: '' - parameters: - - {key: '', value: ''} - person_id: '' - reserved_until: '' - revision: '' - started_at: '' - status: '' - submitted_at: '' - type: '' - version: v1 diff --git a/resource_type/listeners.yaml b/resource_type/listeners.yaml deleted file mode 100644 index 19caa3f..0000000 --- a/resource_type/listeners.yaml +++ /dev/null @@ -1,12 +0,0 @@ -type: listener -path: /listeners -versions: - - version: v0 - prototype: - id: "" - type: "" - revision: "" - listen_on_type: "" - notify_of_new: false - listen_on_all: false - listen_on: [""] diff --git a/resource_type/notifications.yaml b/resource_type/notifications.yaml deleted file mode 100644 index 9660282..0000000 --- a/resource_type/notifications.yaml +++ /dev/null @@ -1,13 +0,0 @@ -type: notification -path: /notifications -versions: - - version: v0 - prototype: - id: "" - type: "" - revision: "" - listener_id: "" - resource_id: "" - resource_revision: "" - resource_change: "" - timestamp: "" diff --git a/resource_type/orgs.yaml b/resource_type/orgs.yaml deleted file mode 100644 index 6566f80..0000000 --- a/resource_type/orgs.yaml +++ /dev/null @@ -1,140 +0,0 @@ -type: org -path: /orgs -versions: - -- version: v0 - prototype: - type: "" - id: "" - revision: "" - -- version: v1 - prototype: - type: "" - id: "" - revision: "" - country: "" - names: [""] - gov_org_ids: - - country: "" - org_id_type: "" - gov_org_id: "" - contacts: - - contact_type: "" - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - is_luotettava_kumppani_member: False - subpaths: - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - -- version: v2 - prototype: - type: "" - id: "" - revision: "" - country: "" - names: [""] - gov_org_ids: - - country: "" - org_id_type: "" - gov_org_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - is_luotettava_kumppani_member: False - subpaths: - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - -- version: v3 - prototype: - type: "" - id: "" - revision: "" - country: "" - names: [""] - gov_org_ids: - - country: "" - org_id_type: "" - gov_org_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - einvoice_operator: "" - einvoice_address: "" - is_luotettava_kumppani_member: False - subpaths: - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - -- version: v4 - prototype: - type: "" - id: "" - revision: "" - country: "" - names: [""] - gov_org_ids: - - country: "" - org_id_type: "" - gov_org_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - einvoice_operator: "" - einvoice_address: "" - is_luotettava_kumppani_member: False - subpaths: - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" diff --git a/resource_type/persons.yaml b/resource_type/persons.yaml deleted file mode 100644 index 7986238..0000000 --- a/resource_type/persons.yaml +++ /dev/null @@ -1,164 +0,0 @@ -type: person -path: /persons -versions: - -- version: v0 - prototype: - type: "" - id: "" - revision: "" - -- version: v1 - prototype: - type: "" - id: "" - revision: "" - names: - - full_name: "" - sort_key: "" - titles: [""] - given_names: [""] - surnames: [""] - subpaths: - photo: - prototype: - body: blob - content_type: "" - private: - prototype: - date_of_birth: "" - gov_ids: - - country: "" - id_type: "" - gov_id: "" - contacts: - - contact_type: "" - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - verification_code: "" - verification_code_expiration_date: "" - email_verification_timestamp: "" - nationalities: [""] - residences: - - country: "" - location: "" - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - files: - - photo - -- version: v2 - prototype: - type: "" - id: "" - revision: "" - names: - - full_name: "" - sort_key: "" - titles: [""] - given_names: [""] - surnames: [""] - subpaths: - photo: - prototype: - body: blob - content_type: "" - private: - prototype: - date_of_birth: "" - gov_ids: - - country: "" - id_type: "" - gov_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - - post_code: "" - post_area: "" - verification_code: "" - verification_code_expiration_date: "" - email_verification_timestamp: "" - nationalities: [""] - residences: - - country: "" - location: "" - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - files: - - photo - -- version: v3 - prototype: - type: "" - id: "" - revision: "" - gluu_user_id: "" - names: - - full_name: "" - sort_key: "" - titles: [""] - given_names: [""] - surnames: [""] - subpaths: - photo: - prototype: - body: blob - content_type: "" - private: - prototype: - date_of_birth: "" - gov_ids: - - country: "" - id_type: "" - gov_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - - post_code: "" - post_area: "" - verification_code: "" - verification_code_expiration_date: "" - email_verification_timestamp: "" - nationalities: [""] - residences: - - country: "" - location: "" - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - files: - - photo diff --git a/resource_type/projects.yaml b/resource_type/projects.yaml deleted file mode 100644 index db62932..0000000 --- a/resource_type/projects.yaml +++ /dev/null @@ -1,21 +0,0 @@ -path: /projects -type: project -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: - id: '' - names: [''] - project_ids: - - {project_id: '', project_id_type: ''} - project_responsible_org: '' - project_responsible_person: '' - revision: '' - type: '' - subpaths: - sync: - prototype: - sync_revision: '' - sync_sources: - - {sync_id: '', sync_source: ''} - version: v1 diff --git a/resource_type/report_accesses.yaml b/resource_type/report_accesses.yaml deleted file mode 100644 index 394b2c2..0000000 --- a/resource_type/report_accesses.yaml +++ /dev/null @@ -1,8 +0,0 @@ -path: /report_accesses -type: report_access -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: {access_time: '', arkisto_id: '', client_id: '', customer_id: '', id: '', - org_id: '', report_id: '', revision: '', type: ''} - version: v1 diff --git a/resource_type/reports.yaml b/resource_type/reports.yaml deleted file mode 100644 index 4163797..0000000 --- a/resource_type/reports.yaml +++ /dev/null @@ -1,17 +0,0 @@ -path: /reports -type: report -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- files: [pdf] - prototype: {generated_timestamp: '', id: '', org: '', report_type: '', revision: '', - tilaajavastuu_status: '', type: ''} - subpaths: - pdf: - prototype: {body: blob, content_type: ''} - sync: - prototype: - sync_revision: '' - sync_sources: - - {sync_id: '', sync_source: ''} - version: v1 diff --git a/resource_type/resource_types.yaml b/resource_type/resource_types.yaml deleted file mode 100644 index 45f5af5..0000000 --- a/resource_type/resource_types.yaml +++ /dev/null @@ -1,11 +0,0 @@ -type: resource_type -path: /resource_types -versions: - -- version: v0 - prototype: - type: "" - id: "" - revision: "" - name: "" - yaml: "" diff --git a/run-qvarn-debug b/run-qvarn-debug deleted file mode 100755 index e272227..0000000 --- a/run-qvarn-debug +++ /dev/null @@ -1,186 +0,0 @@ -#!/bin/sh -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -set -eu - -default_scopes=" -uapi_persons_get -uapi_persons_id_get -uapi_persons_post -uapi_subjects_get -uapi_subjects_id_get -uapi_subjects_post -uapi_version_get - -uapi_events_post -uapi_events_get -uapi_events_id_get -uapi_events_id_put -uapi_events_id_delete -uapi_files_post -uapi_files_get -uapi_files_id_get -uapi_files_search_id_get -uapi_files_id_put -uapi_files_id_delete -uapi_files_id_file_get -uapi_files_id_file_put -uapi_jobs_post -uapi_jobs_get -uapi_jobs_id_get -uapi_jobs_search_id_get -uapi_jobs_id_put -uapi_jobs_id_delete -uapi_contracts_post -uapi_contracts_get -uapi_contracts_id_get -uapi_contracts_search_id_get -uapi_contracts_id_put -uapi_contracts_id_delete -uapi_contracts_id_document_get -uapi_contracts_id_document_put -uapi_persons_post -uapi_persons_get -uapi_persons_id_get -uapi_persons_search_id_get -uapi_persons_id_put -uapi_persons_id_delete -uapi_persons_id_private_get -uapi_persons_id_private_put -uapi_persons_id_photo_get -uapi_persons_id_photo_put -uapi_report_accesses_post -uapi_report_accesses_get -uapi_report_accesses_id_get -uapi_report_accesses_search_id_get -uapi_report_accesses_id_put -uapi_report_accesses_id_delete -uapi_report_accesses_id_pdf_get -uapi_report_accesses_id_pdf_put -uapi_data_cache_post -uapi_data_cache_get -uapi_data_cache_id_get -uapi_data_cache_search_id_get -uapi_data_cache_id_put -uapi_data_cache_id_delete -uapi_projects_post -uapi_projects_get -uapi_projects_id_get -uapi_projects_search_id_get -uapi_projects_id_put -uapi_projects_id_delete -uapi_competence_types_post -uapi_competence_types_get -uapi_competence_types_id_get -uapi_competence_types_search_id_get -uapi_competence_types_id_put -uapi_competence_types_id_delete -uapi_competence_types_id_card_front_get -uapi_competence_types_id_card_front_put -uapi_competence_types_id_card_back_get -uapi_competence_types_id_card_back_put -uapi_competence_types_id_registry_logo_get -uapi_competence_types_id_registry_logo_put -uapi_reports_post -uapi_reports_get -uapi_reports_id_get -uapi_reports_search_id_get -uapi_reports_id_put -uapi_reports_id_delete -uapi_reports_id_pdf_get -uapi_reports_id_pdf_put -uapi_competences_post -uapi_competences_get -uapi_competences_id_get -uapi_competences_search_id_get -uapi_competences_id_put -uapi_competences_id_delete -uapi_version_post -uapi_version_get -uapi_version_id_get -uapi_version_search_id_get -uapi_version_id_put -uapi_version_id_delete -uapi_bolagsfakta_suppliers_post -uapi_bolagsfakta_suppliers_get -uapi_bolagsfakta_suppliers_id_get -uapi_bolagsfakta_suppliers_search_id_get -uapi_bolagsfakta_suppliers_id_put -uapi_bolagsfakta_suppliers_id_delete -uapi_cards_post -uapi_cards_get -uapi_cards_id_get -uapi_cards_search_id_get -uapi_cards_id_put -uapi_cards_id_delete -uapi_cards_id_holder_photo_get -uapi_cards_id_holder_photo_put -uapi_cards_id_issuer_logo_get -uapi_cards_id_issuer_logo_put -uapi_orgs_post -uapi_orgs_get -uapi_orgs_id_get -uapi_orgs_search_id_get -uapi_orgs_id_put -uapi_orgs_id_delete -" - -tmp="$(mktemp -d)" - -cleanup() -{ - rm -rf "$tmp" -} - -trap cleanup EXIT - -token="$1" -case "$#" in - 2) key="$2" ;; - 1) key="$tmp/key" ;; -esac - - - -ISS=test -AUD=aud -./generate-rsa-key "$key" -./create-token "$key" "$ISS" "$AUD" "$default_scopes" > "$token" - -cat <<EOF > "$tmp/qvarn.yaml" -baseurl: http://localhost:12765 -log: - - filename: qvarn.log -token-issuer: $ISS -token-audience: $AUD -token-public-key: $(cat "$key.pub") -resource-type-dir: resource_type -memory-database: yes -database: - host: localhost - port: 5432 - database: qvarn - user: qvarn - password: pass - min_conn: 1 - max_conn: 1 -EOF -export QVARN_CONFIG="$tmp/qvarn.yaml" - -export PYTHONPATH=/home/liw/apifw -gunicorn3 --bind 127.0.0.1:12765 -p "$tmp/pid" -w1 --log-file g.log \ - --log-level debug \ - qvarn.backend:app diff --git a/qvarn/timestamp.py b/salami/__init__.py index 355bee0..f3b31d6 100644 --- a/qvarn/timestamp.py +++ b/salami/__init__.py @@ -14,12 +14,13 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. -import time - - -def get_current_timestamp(): - t = time.time() - tm = time.gmtime(t) - ss = t - int(t) - secs = '%f' % ss - return time.strftime('%Y-%m-%dT%H:%M:%S', tm) + secs[1:] +from .version import __version__, __version_info__ +from .responses import ( + bad_request_response, + created_response, + ok_response, +) +from .log_setup import setup_logging, log +from .router import Router +from .version_router import VersionRouter +from .api import SalamiAPI diff --git a/salami/api.py b/salami/api.py new file mode 100644 index 0000000..ce839d9 --- /dev/null +++ b/salami/api.py @@ -0,0 +1,31 @@ +# Copyright (C) 2017 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + + +import salami + + +class SalamiAPI: + + def __init__(self): + pass + + def find_missing_route(self, path): + salami.log.log('info', msg_text='find_missing_route', path=path) + + if path == '/version': + salami.log.log('info', msg_text='Add /version route') + v = salami.VersionRouter() + return v.get_routes() diff --git a/salami/backend.py b/salami/backend.py new file mode 100644 index 0000000..6412cb3 --- /dev/null +++ b/salami/backend.py @@ -0,0 +1,78 @@ +#!/usr/bin/python3 +# Copyright (C) 2017 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + + +import os + +import apifw +import slog +import yaml + +import salami + + +DEFAULT_CONFIG_FILE = '/dev/null' + + +def dict_logger(log, stack_info=None): + salami.log.log(exc_info=stack_info, **log) + + +def read_config(filename): + with open(filename) as f: + return yaml.safe_load(f) + + +def check_config(cfg): + for key in cfg: + if cfg[key] is None: + raise Exception('Configration %s should not be None' % key) + + +_counter = slog.Counter() + + +def counter(): + new_context = 'HTTP transaction {}'.format(_counter.increment()) + salami.log.set_context(new_context) + + +default_config = { + 'token-public-key': None, + 'token-audience': None, + 'token-issuer': None, + 'log': [], +} + + +config_filename = os.environ.get('SALAMI_CONFIG', DEFAULT_CONFIG_FILE) +actual_config = read_config(config_filename) +config = dict(default_config) +config.update(actual_config or {}) +check_config(config) +salami.setup_logging(config) +salami.log.log('info', msg_text='Salami starting') + +api = salami.SalamiAPI() +app = apifw.create_bottle_application(api, counter, dict_logger, config) + +# If we are running this program directly with Python, and not via +# gunicorn, we can use the Bottle built-in debug server, which can +# make some things easier to debug. + +if __name__ == '__main__': + print('running in debug mode') + app.run(host='127.0.0.1', port=12765) diff --git a/qvarn/logging.py b/salami/log_setup.py index 67f3350..e32137c 100644 --- a/qvarn/logging.py +++ b/salami/log_setup.py @@ -43,21 +43,7 @@ slog.hijack_logging(log, logger_names=gunicorn_loggers) def setup_logging(config): for target in config.get('log', []): - setup_logging_to_target(target) - - -def setup_logging_to_target(target): - rule = get_filter_rules(target) - if 'filename' in target: - setup_logging_to_file(target, rule) - else: - raise Exception('Do not understand logging target %r' % target) - - -def get_filter_rules(target): - if 'filter' in target: - return slog.construct_log_filter(target['filter']) - return slog.FilterAllow() + setup_logging_to_file(target, slog.FilterAllow()) def setup_logging_to_file(target, rule): diff --git a/qvarn/responses.py b/salami/responses.py index eea78d7..138ba28 100644 --- a/qvarn/responses.py +++ b/salami/responses.py @@ -37,10 +37,6 @@ def ok_response(body, headers=None): return response(apifw.HTTP_OK, body, headers) -def no_such_resource_response(msg): - return response(apifw.HTTP_NOT_FOUND, msg, {}) - - def created_response(body, location): headers = { 'Content-Type': 'application/json', @@ -54,44 +50,3 @@ def bad_request_response(body): 'Content-Type': 'text/plain', } return response(apifw.HTTP_BAD_REQUEST, body, headers) - - -def need_sort_response(): - headers = { - 'Content-Type': 'application/json', - } - body = { - 'message': 'LIMIT and OFFSET can only be used with together SORT.', - 'error_code': 'LimitWithoutSortError', - } - return response(apifw.HTTP_BAD_REQUEST, body, headers) - - -def search_parser_error_response(e): - headers = { - 'Content-Type': 'application/json', - } - body = { - 'message': 'Could not parse search condition', - 'error_code': 'BadSearchCondition', - } - return response(apifw.HTTP_BAD_REQUEST, body, headers) - - -def unknown_search_field_response(e): - headers = { - 'Content-Type': 'application/json', - } - body = { - 'field': e.field, - 'message': 'Resource does not contain given field', - 'error_code': 'FieldNotInResource', - } - return response(apifw.HTTP_BAD_REQUEST, body, headers) - - -def conflict_response(body): - headers = { - 'Content-Type': 'text/plain', - } - return response(apifw.HTTP_CONFLICT, body, headers) diff --git a/qvarn/router.py b/salami/router.py index 9f171b0..9f171b0 100644 --- a/qvarn/router.py +++ b/salami/router.py diff --git a/salami/version.py b/salami/version.py new file mode 100644 index 0000000..4e6afe5 --- /dev/null +++ b/salami/version.py @@ -0,0 +1,2 @@ +__version__ = "0.0+git" +__version_info__ = (0, 0, '+git') diff --git a/qvarn/version_router.py b/salami/version_router.py index b9f1056..bc60d5c 100644 --- a/qvarn/version_router.py +++ b/salami/version_router.py @@ -14,10 +14,10 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. -import qvarn +import salami -class VersionRouter(qvarn.Router): +class VersionRouter(salami.Router): def get_routes(self): return [ @@ -31,12 +31,6 @@ class VersionRouter(qvarn.Router): def _version(self, *args, **kwargs): version = { - 'api': { - 'version': qvarn.__version__, - }, - 'implementation': { - 'name': 'Qvarn', - 'version': qvarn.__version__, - }, + 'version': salami.__version__, } - return qvarn.ok_response(version) + return salami.ok_response(version) diff --git a/scripts/build-and-deploy b/scripts/build-and-deploy deleted file mode 100755 index f26f250..0000000 --- a/scripts/build-and-deploy +++ /dev/null @@ -1,76 +0,0 @@ -#!/bin/sh -# -# scripts/build-and-deploy - build Qvarn and upgrade it on given host -# -# Copyright 2016-2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -# This program builds Qvarn and installs it on the host given on the -# command line as an upgrade. The host must already have been -# configured to run Qvarn. This script does nothing to update -# configuration files, for example. -# -# This script is meant to be used while developing Qvarn: make -# changes, commit them, build, deploy to a server, and then test the -# server. -# -# Usage: -# -# $0 HOST BOOL -# -# where HOST is the address of the server on which to upgrade Qvarn, -# and BOOL is "true" to indicate the databases should be emptied -# (recreated), and log files deleted, and "false" if not. -# -# Note that changes must be committed to git before running this -# script, and you must be able to ssh into the server and run sudo -# there. - - -set -eux - -[ -d .git ] -[ -e qvarn/__init__.py ] - -host="$1" -cleandb="$2" - -debian_version="$(dpkg-parsechangelog | sed -n '/^Version: /s///p')" -upstream_version="$(echo "$debian_version" | sed 's/-.*//')" -package="qvarn-jsonb" - -rm -f ../qvarn*_* - -git archive HEAD | xz > "../${package}_${upstream_version}.orig.tar.xz" - -if [ -f "/etc/debian_version" ]; then - debuild -us -uc -else - debuild -d -us -uc -fi - -deb="${package}_${debian_version}_all.deb" -scp "../$deb" "$host:" - -ssh "$host" sudo systemctl stop qvarn || true -ssh "$host" sudo dpkg -i "$deb" -if [ "$cleandb" = true ] -then - ssh "$host" sudo -u postgres dropdb qvarn || true - ssh "$host" sudo -u postgres createdb -O qvarn -E UTF8 -T template0 qvarn - ssh "$host" sudo find /var/log/qvarn -mindepth 1 -type f -delete -fi -ssh "$host" sudo systemctl restart haproxy -ssh "$host" sudo systemctl restart qvarn diff --git a/scripts/pgdump b/scripts/pgdump deleted file mode 100755 index 61b2994..0000000 --- a/scripts/pgdump +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/sh - -set -eu - -sudo -u postgres pg_dump qvarn diff --git a/scripts/pgempty b/scripts/pgempty deleted file mode 100755 index f0962a0..0000000 --- a/scripts/pgempty +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/sh - -set -eu - -sudo -u postgres dropdb qvarn || true -sudo -u postgres createdb -O qvarn -E UTF8 -T template0 qvarn @@ -1,8 +1,6 @@ #!/usr/bin/python3 # -# setup.py - standard Python build-and-package program -# -# Copyright 2017 Vincent Sanders <vince@qvarnlabs.com> +# Copyright 2017 Lars Wirzenius <liw@qvarnlabs.com> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as published by @@ -20,14 +18,14 @@ from setuptools import setup -import qvarn +import salami setup( - name='qvarn-jsonb', - version=qvarn.__version__, - description='backend service for JSON and binary data storage', + name='salami', + version=salami.__version__, + description='thing', author='Lars Wirzenius', author_email='liw@qvarnlabs.com', - packages=['qvarn'], + packages=['salami'], ) diff --git a/start_debug_salami b/start_debug_salami new file mode 100755 index 0000000..7ca166b --- /dev/null +++ b/start_debug_salami @@ -0,0 +1,56 @@ +#!/bin/sh +# Copyright (C) 2017 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +set -eu + +default_scopes=" +uapi_version_get +" + +tmp="$(mktemp -d)" + +cleanup() +{ + rm -rf "$tmp" +} + +trap cleanup EXIT + +token="$1" +case "$#" in + 2) key="$2" ;; + 1) key="$tmp/key" ;; +esac + + + +ISS=test +AUD=aud +./generate-rsa-key "$key" +./create-token "$key" "$ISS" "$AUD" "$default_scopes" > "$token" + +cat <<EOF > "$tmp/salami.yaml" +log: + - filename: salami.log +token-issuer: $ISS +token-audience: $AUD +token-public-key: $(cat "$key.pub") +EOF +export SALAMI_CONFIG="$tmp/salami.yaml" + +gunicorn3 --bind 127.0.0.1:12765 -p "$tmp/pid" -w1 --log-file g.log \ + --log-level debug \ + salami.backend:app diff --git a/start_qvarn b/start_salami index d446843..96dce90 100755 --- a/start_qvarn +++ b/start_salami @@ -16,11 +16,11 @@ set -eu -export QVARN_CONFIG="/etc/qvarn/qvarn.conf" +export START_CONFIG="/etc/salami/salami.yaml" gunicorn3 \ --bind 127.0.0.1:12765 \ -w1 \ --log-file /var/log/qvarn/gunicorn3.log \ --log-level debug \ - qvarn.backend:app + salami.backend:app diff --git a/without-tests b/without-tests index a752d9d..fac0e45 100644 --- a/without-tests +++ b/without-tests @@ -1,16 +1,9 @@ setup.py -qvarn/__init__.py -qvarn/api_errors.py -qvarn/backend.py -qvarn/file_router.py -qvarn/logging.py -qvarn/notification_router.py -qvarn/resource_router.py -qvarn/responses.py -qvarn/router.py -qvarn/sql.py -qvarn/subresource_router.py -qvarn/timestamp.py -qvarn/version.py -qvarn/version_router.py -yarns/lib.py +salami/__init__.py +salami/api.py +salami/backend.py +salami/log_setup.py +salami/responses.py +salami/router.py +salami/version.py +salami/version_router.py diff --git a/yarns/900-implements.yarn b/yarns/900-implements.yarn deleted file mode 100644 index 2f7ee0f..0000000 --- a/yarns/900-implements.yarn +++ /dev/null @@ -1,272 +0,0 @@ -# Step implementations - -This chapter shows the scenario step implementations. - -## Start and stop Qvarn - -Start a Qvarn running in the background. - - IMPLEMENTS GIVEN a running qvarn instance - import os, time, cliapp, yaml, yarnutils - privkey, pubkey = create_token_signing_key_pair() - print repr(privkey) - print repr(pubkey) - assert privkey - assert pubkey - open('key', 'w').write(privkey) - vars['aud'] = 'http://api.test.example.com' - vars['iss'] = 'qvarn.yarn' - vars['privkey'] = privkey - vars['pubkey'] = pubkey - vars['api.log'] = 'qvarn.log' - vars['gunicorn3.log'] = 'gunicorn3.log' - vars['pid-file'] = 'pid' - vars['port'] = cliapp.runcmd([os.path.join(srcdir, 'randport' )]).strip() - vars['url'] = 'http://127.0.0.1:{}'.format(vars['port']) - vars['API_URL'] = vars['url'] - config = { - 'log': [ - { - 'filename': vars['api.log'], - }, - ], - 'baseurl': vars['url'], - 'token-issuer': vars['iss'], - 'token-audience': vars['aud'], - 'token-public-key': vars['pubkey'], - 'resource-type-dir': os.path.join(srcdir, 'resource_type'), - } - config = add_postgres_config(config) - env = dict(os.environ) - env['QVARN_CONFIG'] = os.path.join(datadir, 'qvarn.yaml') - yaml.safe_dump(config, open(env['QVARN_CONFIG'], 'w')) - argv = [ - 'gunicorn3', - '--daemon', - '--bind', '127.0.0.1:{}'.format(vars['port']), - '-p', vars['pid-file'], - 'qvarn.backend:app', - ] - cliapp.runcmd(argv, env=env, stdout=None, stderr=None) - until = time.time() + 2.0 - while time.time() < until and not os.path.exists(vars['pid-file']): - time.sleep(0.01) - assert os.path.exists(vars['pid-file']) - -## Stop a Qvarn we started - - IMPLEMENTS FINALLY qvarn is stopped - import os, signal, yarnutils - pid = int(cat(vars['pid-file'])) - os.kill(pid, signal.SIGTERM) - -## API requests of various kinds - - IMPLEMENTS WHEN client requests GET (/.+) without token - path = get_next_match() - path = expand_vars(path, vars) - vars['status_code'], vars['headers'], vars['body'] = get(vars['url'] + path) - - IMPLEMENTS WHEN client requests GET (/.+) using token - path = get_next_match() - path = expand_vars(path, vars) - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - } - vars['status_code'], vars['headers'], vars['body'] = get( - vars['url'] + path, headers) - - IMPLEMENTS WHEN client requests POST (/.+) with token and body (.+) - path = get_next_match() - body = get_next_match() - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - 'Content-Type': 'application/json', - } - vars['status_code'], vars['headers'], vars['body'] = post( - vars['url'] + path, headers=headers, body=body) - - IMPLEMENTS WHEN client requests PUT (/.+) with token and body (.+) - path = get_next_match() - path = expand_vars(path, vars) - body = get_next_match() - body = expand_vars(body, vars) - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - 'Content-Type': 'application/json', - } - vars['status_code'], vars['headers'], vars['body'] = put( - vars['url'] + path, headers=headers, body=body) - - IMPLEMENTS WHEN client requests PUT (/[a-z0-9/${}]+) with token, revision (\S+), content-type (\S+), and empty body - path = expand_vars(get_next_match(), vars) - revision = expand_vars(get_next_match(), vars) - ctype = expand_vars(get_next_match(), vars) - body = '' - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - 'Revision': revision, - 'Content-Type': ctype, - } - vars['status_code'], vars['headers'], vars['body'] = put( - vars['url'] + path, headers=headers, body=body) - - IMPLEMENTS WHEN client requests PUT (/[a-z0-9/${}]+) with token, revision (\S+), content-type (\S+), and body "(.+)" - path = expand_vars(get_next_match(), vars) - revision = expand_vars(get_next_match(), vars) - ctype = expand_vars(get_next_match(), vars) - body = unescape(expand_vars(get_next_match(), vars)) - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - 'Revision': revision, - 'Content-Type': ctype, - } - vars['status_code'], vars['headers'], vars['body'] = put( - vars['url'] + path, headers=headers, body=body) - - IMPLEMENTS WHEN client requests DELETE (/.+) with token - path = get_next_match() - path = expand_vars(path, vars) - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - } - vars['status_code'], vars['headers'], vars['body'] = delete( - vars['url'] + path, headers=headers) - - IMPLEMENTS WHEN client uploads a fake jpg - assert 0 - -## API access token creation - - IMPLEMENTS WHEN client gets an authorization token with scope "(.+)" - scopes = get_next_match() - print 'privkey', repr(vars['privkey']) - assert vars['privkey'] - vars['token'] = create_token(vars['privkey'], vars['iss'], vars['aud'], scopes) - -## UUID creation - - IMPLEMENTS GIVEN unique random identifier (\S+) - import uuid - name = get_next_match() - vars[name] = str(uuid.uuid4()) - -## API request result checking - - IMPLEMENTS THEN HTTP status code is (\d+) (.*) - expected = int(get_next_match()) - assertEqual(vars['status_code'], expected) - - IMPLEMENTS THEN HTTP (\S+) header is (.+) - header = get_next_match() - value = expand_vars(get_next_match(), vars) - assertEqual(vars['headers'].get(header), value) - - IMPLEMENTS THEN remember HTTP (\S+) header as (.+) - header = get_next_match() - name = get_next_match() - vars[name] = vars['headers'].get(header) - - IMPLEMENTS THEN resource id is (\S+) - import json - name = get_next_match() - print 'body:', repr(vars['body']) - body = json.loads(vars['body']) - vars[name] = body['id'] - - IMPLEMENTS THEN revision is (\S+) - import json - name = get_next_match() - body = json.loads(vars['body']) - vars[name] = body['revision'] - - IMPLEMENTS THEN revisions (\S+) and (\S+) are different - rev1 = get_next_match() - rev2 = get_next_match() - assertNotEqual(vars[rev1], vars[rev2]) - - IMPLEMENTS THEN revisions (\S+) and (\S+) match - rev1 = get_next_match() - rev2 = get_next_match() - assertEqual(vars[rev1], vars[rev2]) - - IMPLEMENTS THEN JSON body matches (.+) - import json - wanted = get_next_match() - print 'wanted1', repr(wanted) - wanted = expand_vars(wanted, vars) - print 'wanted2', repr(wanted) - wanted = json.loads(wanted) - actual = json.loads(vars['body']) - print 'actual ', repr(actual) - print 'wanted3', repr(wanted) - assertTrue(values_match(wanted, actual)) - - IMPLEMENTS THEN body is "(.+)" - wanted = unescape(expand_vars(get_next_match(), vars)) - body = vars['body'] - assertTrue(values_match(wanted, body)) - - IMPLEMENTS THEN search result contains (.+) - import json - wanted1 = get_next_match() - wanted2 = expand_vars(wanted1, vars) - wanted = json.loads(wanted2) - actual = json.loads(vars['body']) - print 'wanted1:', repr(wanted1) - print 'wanted2:', repr(wanted2) - print 'wanted:', repr(wanted) - print 'actual:', repr(actual) - assertTrue(actual['resources']) - found = False - for result in actual['resources']: - if values_match(wanted, result): - print 'MATCH!', repr(wanted), repr(result) - found = True - break - print 'no match', repr(wanted), repr(result) - assertTrue(found) - - IMPLEMENTS THEN search result does NOT contain (.+) - import json - wanted1 = get_next_match() - wanted2 = expand_vars(wanted1, vars) - wanted = json.loads(wanted2) - actual = json.loads(vars['body']) - print 'wanted1:', repr(wanted1) - print 'wanted2:', repr(wanted2) - print 'wanted:', repr(wanted) - print 'actual:', repr(actual) - found = False - for result in actual['resources']: - if values_match(wanted, result): - found = True - assertFalse(found) - - IMPLEMENTS THEN search result at index (\d+) has id (\S+) - import json - index = int(get_next_match()) - id_name = get_next_match() - body = json.loads(vars['body']) - print 'body', repr(body) - resources = body['resources'] - print 'resources', repr(resources) - print 'len resources', len(resources) - print 'index', index - assert index < len(resources) - obj = resources[index] - print 'resource at index', repr(obj) - print 'id', repr(obj['id']) - vars[id_name] = obj['id'] - - IMPLEMENTS THEN search result has (\d+) resources - wanted = int(get_next_match()) - body = json.loads(vars['body']) - print 'body', repr(body) - resources = body['resources'] - print 'resources', repr(resources) - print 'len resources', len(resources) - assertEqual(wanted, len(resources)) - - IMPLEMENTS THEN response has header WWW-Authenticate containing "(.+)" - assert 0 diff --git a/yarns/lib.py b/yarns/lib.py deleted file mode 100644 index 459a821..0000000 --- a/yarns/lib.py +++ /dev/null @@ -1,157 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import json -import os -import re -import tempfile - - -import cliapp -import Crypto.PublicKey.RSA -import requests -import yaml - - -from yarnutils import * - - -srcdir = os.environ['SRCDIR'] -datadir = os.environ['DATADIR'] - - -vars = Variables(datadir) - - -def hexdigit(c): - return ord(c) - ord('0') - - -def unescape(s): - t = '' - while s: - if s.startswith('\\x') and len(s) >= 4: - a = hexdigit(s[2]) - b = hexdigit(s[3]) - t += chr(a * 16 + b) - s = s[4:] - else: - t += s[0] - s = s[1:] - return t - - -def add_postgres_config(config): - pg = os.environ.get('QVARN_POSTGRES') - if pg: - with open(pg) as f: - config['database'] = yaml.safe_load(f) - config['memory-database'] = False - return config - - -def get(url, headers=None): - print('get: url={} headers={}'.format(url, headers)) - r = requests.get(url, headers=headers) - return r.status_code, dict(r.headers), r.content - - -def post(url, headers=None, body=None): - r = requests.post(url, headers=headers, data=body) - return r.status_code, dict(r.headers), r.text - - -def put(url, headers=None, body=None): - r = requests.put(url, headers=headers, data=body) - return r.status_code, dict(r.headers), r.text - - -def delete(url, headers=None): - r = requests.delete(url, headers=headers) - return r.status_code, dict(r.headers), r.text - - -def create_token_signing_key_pair(): - RSA_KEY_BITS = 4096 # A nice, currently safe length - key = Crypto.PublicKey.RSA.generate(RSA_KEY_BITS) - return key.exportKey('PEM'), key.exportKey('OpenSSH') - - -def create_token(privkey, iss, aud, scopes): - filename = write_temp(privkey) - argv = [ - os.path.join(srcdir, 'create-token'), - filename, - iss, - aud, - scopes, - ] - return cliapp.runcmd(argv) - - -def cat(filename): - return open(filename).read() - - -def write_temp(data): - fd, filename = tempfile.mkstemp(dir=datadir) - os.write(fd, data) - os.close(fd) - return filename - - -def expand_vars(text, vars): - result = '' - while text: - m = re.search(r'\${(?P<name>[^}]+)}', text) - if not m: - result += text - break - name = m.group('name') - print('expanding ', name) - result += text[:m.start()] + vars[name] - text = text[m.end():] - return result - - -def values_match(wanted, actual): - print - print 'wanted:', repr(wanted) - print 'actual:', repr(actual) - - if type(wanted) != type(actual): - print 'wanted and actual types differ', type(wanted), type(actual) - return False - - if isinstance(wanted, dict): - for key in wanted: - if key not in actual: - print 'key {!r} not in actual'.format(key) - return False - if not values_match(wanted[key], actual[key]): - return False - elif isinstance(wanted, list): - if len(wanted) != len(actual): - print 'wanted and actual are of different lengths' - for witem, aitem in zip(wanted, actual): - if not values_match(witem, aitem): - return False - else: - if wanted != actual: - print 'wanted and actual differ' - return False - - return True diff --git a/yarns/smoke.yarn b/yarns/smoke.yarn deleted file mode 100644 index 6e774d2..0000000 --- a/yarns/smoke.yarn +++ /dev/null @@ -1,1204 +0,0 @@ ---- -title: qvarn-jsonb integration tests -author: Lars Wirzenius / QvarnLabs Ab -date: work in progress -... - - -# Introduction - -This is an integration test suite for the Qvarn HTTP API. - -## History and background - -Qvarn started as an internal project in 2015 at Suomen Tilaajavastuu -(https://www.tilaajavastuu.fi/en/), a company that provides various -reproting services to the Finnish construction industry. Later that -year the Swedish Construction Federation -(https://www.sverigesbyggindustrier.se/english) adopted Qvarn as a -platform for ID06 identity card services in Sweden. From the start, a -goal for Qvarn has been to make it easy to comply with the EU General -Data Protection Regulation. - -In 2016 Qvarn spun off to its own company, QvarnLabs Ab, which -provides consulting, development, and support services around Qvarn. - - -# Version checking - - SCENARIO Qvarn reports its version - - GIVEN a running qvarn instance - - WHEN client requests GET /version without token - THEN HTTP status code is 200 OK - - WHEN client gets an authorization token with scope "uapi_version_get" - AND client requests GET /version using token - THEN HTTP status code is 200 OK - - FINALLY qvarn is stopped - - -# Manage a subject - - SCENARIO user manages a subject resource - - GIVEN a running Qvarn instance - - WHEN client requests GET /subjects without token - THEN HTTP status code is 401 Unauthorized - - WHEN client requests GET /subjects/notexist without token - THEN HTTP status code is 401 Unauthorized - - WHEN client gets an authorization token with scope - ... "uapi_subjects_get uapi_subjects_post uapi_subjects_id_get - ... uapi_subjects_id_put uapi_subjects_id_delete" - AND client requests GET /subjects using token - THEN HTTP status code is 200 OK - - WHEN client requests GET /subjects using token - THEN HTTP status code is 200 OK - AND search result does NOT contain { "id": "subject" } - - WHEN client requests POST /subjects with token and body - ... { - ... "type": "subject", - ... "names": [ - ... { "full_name": "Jason Bourne" } - ... ] - ... } - THEN HTTP status code is 201 Created - AND resource id is ID1 - AND revision is REV1 - - WHEN client requests GET /subjects/${ID1} without token - THEN HTTP status code is 401 Unauthorized - - WHEN client requests GET /subjects/${ID1} using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { "revision": "${REV1}", "id": "${ID1}", "type": "subject", - ... "names": [{ "full_name": "Jason Bourne" }] } - -For silly hysterical raisins, the Qvarn API is defined to return 400 -if updating a resource with the wrong revision in the body, instead -of 404. We may want to fix this some day. - - WHEN client requests PUT /subjects/${ID1} with token and body - ... { "id": "wrong", "revision": "${REV1}", "names": [{"full_name": "Dave Webb"}]} - THEN HTTP status code is 400 Bad - - WHEN client requests PUT /subjects/${ID1} with token and body - ... { "id": "${ID1}", "revision": "${REV1}", "names": [{"full_name": "Dave Webb"}]} - THEN HTTP status code is 200 OK - AND revision is REV2 - - WHEN client requests PUT /subjects/${ID1} with token and body - ... { "type": "subject", "id": "${ID1}", "revision": "${REV2}", - ... "names": [{"full_name": "David Webb" }]} - THEN HTTP status code is 200 OK - AND revision is REV3 - - WHEN client requests GET /subjects/${ID1} using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { "revision": "${REV3}", "id": "${ID1}", "type": "subject", - ... "names": [{"full_name": "David Webb" }]} - - WHEN client requests DELETE /subjects/${ID1} with token - THEN HTTP status code is 200 OK - - WHEN client requests GET /subjects/${ID1} using token - THEN HTTP status code is 404 Not Found - - FINALLY qvarn is stopped - - -# Manage notifications - -Notifications are a special resource type. Notifications created -automatically by Qvarn, but the API client may delete them. - - SCENARIO manage notifications - - GIVEN a running Qvarn instance - -Client has needed access rights for orgs resource. - - WHEN client gets an authorization token with scope - ... "uapi_orgs_listeners_post uapi_orgs_listeners_id_get - ... uapi_orgs_listeners_get uapi_orgs_listeners_id_notifications_get - ... uapi_orgs_post uapi_orgs_listeners_id_notifications_id_get - ... uapi_orgs_listeners_id_put uapi_orgs_id_put uapi_orgs_id_delete - ... uapi_orgs_listeners_id_delete - ... uapi_orgs_listeners_id_notifications_id_delete" - - WHEN client requests POST /orgs/listeners with token and body - ... { - ... "notify_of_new": true - ... } - THEN HTTP status code is 201 Created - AND JSON body matches - ... { - ... "type": "listener", - ... "notify_of_new": true, - ... "listen_on": [] - ... } - AND resource id is LISTENID1 - AND HTTP Location header is ${API_URL}/orgs/listeners/${LISTENID1} - - WHEN client requests POST /orgs/listeners with token and body - ... { - ... "notify_of_new": false - ... } - THEN HTTP status code is 201 Created - AND JSON body matches - ... { - ... "type": "listener", - ... "notify_of_new": false, - ... "listen_on": [] - ... } - AND resource id is LISTENID2 - AND HTTP Location header is ${API_URL}/orgs/listeners/${LISTENID2} - AND revision is REV1 - - WHEN client requests POST /orgs/listeners with token and body - ... { - ... "notify_of_new": false, - ... "listen_on_all": true - ... } - THEN HTTP status code is 201 Created - AND JSON body matches - ... { - ... "type": "listener", - ... "notify_of_new": false, - ... "listen_on_all": true, - ... "listen_on": [] - ... } - AND resource id is LISTENID3 - AND HTTP Location header is ${API_URL}/orgs/listeners/${LISTENID3} - - WHEN client requests GET /orgs/listeners/${LISTENID1} using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "id": "${LISTENID1}", - ... "type": "listener", - ... "notify_of_new": true, - ... "listen_on": [] - ... } - - WHEN client requests GET /orgs/listeners using token - THEN HTTP status code is 200 OK - THEN search result contains {"id": "${LISTENID1}"} - THEN search result contains {"id": "${LISTENID2}"} - -A listener has no notifications initially. - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [] - ... } - - WHEN client requests POST /orgs with token and body - ... { - ... "names": ["Universal Exports"] - ... } - THEN resource id is ORGID1 - AND revision is REV2 - - WHEN client requests POST /orgs with token and body - ... { - ... "names": ["Telebulvania Ltd"] - ... } - THEN resource id is ORGID2 - -After adding the new organizations the first listener should be notified while -the second and third should have no notifications. - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result at index 0 has id MSGID1 - AND search result at index 1 has id MSGID2 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications/${MSGID1} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID1}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_change": "created" - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications/${MSGID2} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID2}", - ... "type": "notification", - ... "resource_id": "${ORGID2}", - ... "resource_change": "created" - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [] - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [] - ... } - -We update the empty listener to listen on organization changes and update the -organization checking for the correct notification to appear. The third -listener listening to all the changes should get the notification, too. - - WHEN client requests PUT /orgs/listeners/${LISTENID2} with token and body - ... { - ... "notify_of_new": false, - ... "listen_on": ["${ORGID1}"], - ... "revision": "${REV1}" - ... } - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "type": "listener", - ... "notify_of_new": false, - ... "listen_on": ["${ORGID1}"] - ... } - - WHEN client requests PUT /orgs/${ORGID1} with token and body - ... { - ... "names": ["Universal Experts"], - ... "revision": "${REV2}" - ... } - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result at index 0 has id MSGID3 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications/${MSGID3} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID3}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_change": "updated" - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result has 1 resources - AND search result at index 0 has id MSGID4 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications/${MSGID4} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID4}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_change": "updated" - ... } - -The first listener gets no additional notifications. - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${MSGID1}"}, {"id": "${MSGID2}"} - ... ] - ... } - -We delete the organization and check for the correct notification to appear. - - WHEN client requests DELETE /orgs/${ORGID1} with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result at index 1 has id MSGID5 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications/${MSGID5} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID5}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_revision": null, - ... "resource_change": "deleted" - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result at index 1 has id MSGID6 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications/${MSGID6} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID6}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_revision": null, - ... "resource_change": "deleted" - ... } - -The first listener gets no additional notifications. - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${MSGID1}"}, {"id": "${MSGID2}"} - ... ] - ... } - -Deletion of a listener deletes also the notifications. - - WHEN client requests DELETE /orgs/listeners/${LISTENID1} with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications/${MSGID1} - ... using token - THEN HTTP status code is 404 Not Found - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications/${MSGID2} - ... using token - THEN HTTP status code is 404 Not Found - - WHEN client requests GET /orgs/listeners/${LISTENID1} using token - THEN HTTP status code is 404 Not Found - -Notification can be deleted. - - WHEN client requests - ... DELETE /orgs/listeners/${LISTENID2}/notifications/${MSGID3} - ... with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications/${MSGID3} - ... using token - THEN HTTP status code is 404 Not Found - - WHEN client requests - ... DELETE /orgs/listeners/${LISTENID3}/notifications/${MSGID4} - ... with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications/${MSGID4} - ... using token - THEN HTTP status code is 404 Not Found - - WHEN client requests - ... DELETE /orgs/listeners/${LISTENID2}/notifications/${MSGID5} - ... with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... DELETE /orgs/listeners/${LISTENID3}/notifications/${MSGID6} - ... with token - THEN HTTP status code is 200 OK - - - FINALLY qvarn is stopped - -# Listeners for one resource type only - -This scenario tests for a problem found in the first public release of -Qvarn (JSONB): all listeners would be reported for all resource types, -intead of only the one they were created for. - - SCENARIO notifications only for the right type - - GIVEN a running qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_persons_post - ... uapi_persons_listeners_post - ... uapi_persons_listeners_get - ... uapi_persons_listeners_id_notifications_get - ... uapi_persons_listeners_id_notifications_id_get - ... uapi_orgs_post - ... uapi_orgs_listeners_post - ... uapi_orgs_listeners_get - ... uapi_orgs_listeners_id_notifications_get - ... uapi_orgs_listeners_id_notifications_id_get - ... " - -Make sure there are no listeners yet. - - WHEN client requests GET /orgs/listeners using token - THEN JSON body matches { "resources": [] } - - WHEN client requests GET /persons/listeners using token - THEN JSON body matches { "resources": [] } - -Create a listeners for orgs. - - WHEN client requests POST /orgs/listeners with token and body - ... { - ... "notify_of_new": true - ... } - THEN resource id is ORGLISTENER - - WHEN client requests GET /orgs/listeners using token - THEN JSON body matches { "resources": [{ "id": "${ORGLISTENER}" }]} - - WHEN client requests GET /persons/listeners using token - THEN JSON body matches { "resources": [] } - -Create a listener for persons. - - WHEN client requests POST /persons/listeners with token and body - ... { - ... "notify_of_new": true - ... } - THEN resource id is PERSONLISTENER - - WHEN client requests GET /orgs/listeners using token - THEN JSON body matches { "resources": [{ "id": "${ORGLISTENER}" }]} - - WHEN client requests GET /persons/listeners using token - THEN JSON body matches { "resources": [{ "id": "${PERSONLISTENER}"}]} - -Create a person, make sure only the correct notifications are created. - - WHEN client requests POST /persons with token and body - ... { - ... "names": [{ "full_name": "James Bond" }] - ... } - THEN resource id is PERSONID - - WHEN client requests - ... GET /orgs/listeners/${ORGLISTENER}/notifications - ... using token - THEN JSON body matches { "resources": [] } - - WHEN client requests - ... GET /persons/listeners/${PERSONLISTENER}/notifications - ... using token - THEN search result at index 0 has id MSGID - - WHEN client requests - ... GET /persons/listeners/${PERSONLISTENER}/notifications/${MSGID} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID}", - ... "type": "notification", - ... "resource_id": "${PERSONID}", - ... "resource_change": "created" - ... } - - FINALLY qvarn is stopped - -# Use subresources - -Subresources are additional resources that are attached to a resource, -and be managed and access controlled separetly. However, they share -existence and revision with their parent: if the parent is deleted, so -is the subresource, and if either the parent or the subresource is -changed, and gets a new revision, so does the other. - -Subresources always exist, if defined in the resource type spec. They -don't need to be created specially. - - SCENARIO manage subresources - - GIVEN a running qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_sub_put uapi_subjects_sub_get" - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Alfred" } ] } - THEN resource id is ID - AND revision is REV - - WHEN client requests GET /subjects/${ID}/sub using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "subfield": "" - ... } - - WHEN client requests PUT /subjects/${ID}/sub with token and body - ... { - ... "revision": "wrong", - ... "subfield": "Steven Segal" - ... } - THEN HTTP status code is 409 Conflict - - WHEN client requests PUT /subjects/${ID}/sub with token and body - ... { - ... "revision": "${REV}", - ... "subfield": "Steven Segal" - ... } - THEN HTTP status code is 200 OK - AND revision is REV2 - AND JSON body matches - ... { - ... "revision": "${REV2}", - ... "subfield": "Steven Segal" - ... } - - FINALLY qvarn is stopped - - -# Manage blobs - -Blobs are like sub-resources, but they're arbitrary binary data, not -JSON. - - SCENARIO manage blobs - - GIVEN a running qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_blob_put uapi_subjects_blog_get" - -Create a subject. - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Alfred" } ] } - THEN resource id is ID - AND revision is REV1 - -Newly created subject does not have a blob. - - WHEN client requests GET /subjects/${ID}/blob using token - THEN HTTP status code is 404 Not found - -Uploading an empty blob doesn't work. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision REV1, - ... content-type image/jpeg, - ... and empty body - THEN HTTP status code is 411 Length required - -Uploading a COMPLETELY VALID JPEG as a blob fails, if subject resource -revision is wrong. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision BADREV, - ... content-type image/jpeg, - ... and body "FAKE JPEG" - THEN HTTP status code is 409 Conflict - -Uploading with valid revision works. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision ${REV1}, - ... content-type image/jpeg, - ... and body "FAKE JPEG" - THEN HTTP status code is 200 OK - -Do we get the right blob back? Also, note that subject revision -should've changed. - - WHEN client requests GET /subjects/${ID}/blob using token - THEN HTTP status code is 200 OK - AND HTTP Content-Type header is image/jpeg - AND body is "FAKE JPEG" - AND remember HTTP Revision header as REV2 - AND revisions REV1 and REV2 are different - -Uploading with old revision fails. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision ${REV1}, - ... content-type image/jpeg, - ... and body "FAKE JPEG" - THEN HTTP status code is 409 Conflict - -Uploading a new blob with the current revision works. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision ${REV2}, - ... content-type image/jpeg, - ... and body "\x89" - THEN HTTP status code is 200 OK - AND remember HTTP Revision header as REV3 - -And it did get updated. - - WHEN client requests GET /subjects/${ID}/blob using token - THEN HTTP status code is 200 OK - AND HTTP Content-Type header is image/jpeg - AND body is "\x89" - -Updating parent doesn't affect the blob. - - WHEN client requests PUT /subjects/${ID} with token and body - ... { - ... "type": "subject", - ... "revision": "${REV3}", - ... "names": [ { "full_name": "Melissa" } ] - ... } - THEN revision is REV4 - - WHEN client requests GET /subjects/${ID}/blob using token - THEN HTTP status code is 200 OK - AND HTTP Content-Type header is image/jpeg - AND body is "\x89" - AND remember HTTP Revision header as REV5 - AND revisions REV4 and REV5 match - - FINALLY qvarn is stopped - -# Search subjects - - SCENARIO search subjects - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_search_id_get - ... uapi_subjects_id_delete" - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Alfred" } ] } - THEN resource id is ID1 - AND revision is REV1 - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Alfred" } ] } - THEN resource id is ID2 - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Bruce" } ] } - THEN resource id is ID3 - - WHEN client requests GET /subjects/search/exact/full_name/Batman - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches { "resources": []} - -Make sure searches are case-insensitive. - - WHEN client requests GET /subjects/search/exact/full_name/ALFRED - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result does NOT contain {"id": "${ID3}"} - - WHEN client requests GET /subjects/search/exact/full_name/bruce - ... using token - THEN HTTP status code is 200 OK - AND search result does NOT contain {"id": "${ID1}"} - AND search result does NOT contain {"id": "${ID2}"} - AND search result contains {"id": "${ID3}"} - AND JSON body matches { "resources": [{"id": "${ID3}"}]} - - WHEN client requests - ... GET /subjects/search/contains/full_name/fred - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result does NOT contain {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/gt/full_name/Alfred - ... using token - THEN HTTP status code is 200 OK - AND search result does NOT contain {"id": "${ID1}"} - AND search result does NOT contain {"id": "${ID2}"} - AND search result contains {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/ge/full_name/Alfred - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result contains {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/startswith/full_name/Alfred - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result does NOT contain {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/show/full_name/contains/full_name/fred - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result does NOT contain {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/show_all/exact/full_name/Bruce - ... using token - THEN HTTP status code is 200 OK - AND search result contains - ... { - ... "id": "${ID3}", - ... "names": [ { "full_name": "Bruce" } ] - ... } - - WHEN client requests - ... GET /subjects/search/exact/full_name/Br%2Fce - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches { "resources": [] } - - GIVEN unique random identifier RANDOM - WHEN client requests PUT /subjects/${ID1}/sub with token and body - ... { "subfield": "${RANDOM}", "revision": "${REV1}" } - THEN HTTP status code is 200 OK - - WHEN client requests GET /subjects/search/exact/subfield/${RANDOM} using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - - FINALLY qvarn is stopped - - -# More search - -This scenario creates two resources, and searches with two conditions. -Each condition matches a resource, but the two match different ones. -Thus the result should be an empty set. - - SCENARIO search with two conditions when two resources match one - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_search_id_get - ... uapi_subjects_id_delete" - - WHEN client requests POST /subjects with token and body - ... { - ... "type": "subject", - ... "names": [{ - ... "full_name": "Clark Kent", - ... "sort_key": "Clark the superman" - ... }] - ... } - THEN resource id is ID1 - - WHEN client requests POST /subjects with token and body - ... { - ... "type": "subject", - ... "names": [{ - ... "full_name": "Clark Celt", - ... "sort_key": "a nobody" - ... }] - ... } - THEN resource id is ID2 - - WHEN client requests - ... GET /subjects/search/contains/full_name/Kent/contains/sort_key/nobody - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches { "resources": [ ]} - - WHEN client requests - ... GET /subjects/search/contains/full_name/Clark - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... {"resources": [{"id": "${ID1}"}, {"id": "${ID2}"}]} - - WHEN client requests - ... GET /subjects/search/contains/full_name/Kent - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches {"resources": [{"id": "${ID1}"}]} - - WHEN client requests - ... GET /subjects/search/contains/sort_key/nobody - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches {"resources": [{"id": "${ID2}"}]} - - FINALLY qvarn is stopped - -# Searching with multiple conditions - - SCENARIO search with multiple conditions - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_contracts_post uapi_contracts_search_id_get" - -Create an organisation, a person and an employment contract between them. - - WHEN client requests POST /contracts with token and body - ... {"contract_type": "employment", - ... "start_date": "2016-01-01", - ... "contract_parties": [ - ... { - ... "type": "org", - ... "resource_id": "org-1", - ... "role": "employer" - ... }, - ... { - ... "type": "person", - ... "resource_id": "person-1", - ... "role": "employee" - ... } - ... ], - ... "contract_state": "active" - ... } - THEN HTTP status code is 201 Created - AND resource id is CONTRACT_ID1 - -Perform searches matching different instances of the same nested element. - - WHEN client requests - ... GET /contracts/search/exact/resource_id/org-1/exact/resource_id/person-1 - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${CONTRACT_ID1}"} - - WHEN client requests - ... GET /contracts/search/exact/resource_id/org-1/exact/resource_id/wrong_id - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [] - ... } - - FINALLY qvarn is stopped - -# Sort search results - - SCENARIO search with sort - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_search_id_get" - -Create several person resources. - - GIVEN unique random identifier UID - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person A", - ... "sort_key": "3, Person", - ... "given_names": ["Test"], - ... "surnames": ["Person", "A"] - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID3 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person B", - ... "sort_key": "1, Person", - ... "given_names": ["Test"], - ... "surnames": ["Person", "B"] - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID1 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person C", - ... "sort_key": "2, Person", - ... "given_names": ["Test"], - ... "surnames": ["Person", "C"] - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID2 - -Search person resources and sort results by `sort_key`. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/sort_key - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID1}"}, - ... {"id": "${ID2}"}, - ... {"id": "${ID3}"} - ... ] - ... } - -Sort person resources using different sort key. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/full_name - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID3}"}, - ... {"id": "${ID1}"}, - ... {"id": "${ID2}"} - ... ] - ... } - -Search person resources and sort results by two search keys, where first search -key is a list containing more than one value. First key is `surnames`, where -each resource has same first surname, and second key is `sort_key`. Since each -first surname is the same, results should fall back to the second sort key. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/surnames/sort/sort_key - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID1}"}, - ... {"id": "${ID2}"}, - ... {"id": "${ID3}"} - ... ] - ... } - -Search with only search operator should also work, returning all available -resource ids. - - WHEN client requests - ... GET /subjects/search/sort/sort_key - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result contains {"id": "${ID3}"} - - FINALLY qvarn is stopped - -# Sort search results with /offset and /limit - - SCENARIO search with /offset and /limit - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_search_id_get" - -Create several person resources. - - GIVEN unique random identifier UID - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 1" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID1 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 2" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID2 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 3" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID3 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 4" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID4 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 5" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID5 - -Sort, return first two hits. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/full_name/limit/2 - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID1}"}, - ... {"id": "${ID2}"} - ... ] - ... } - -Sort, return second set of two hits. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/full_name/offset/2/limit/2 - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID3}"}, - ... {"id": "${ID4}"} - ... ] - ... } - -Sort, return third set of two hits, which is actualy only one item. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/full_name/offset/4/limit/2 - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID5}"} - ... ] - ... } - -Don't sort. Then /offset and /limit are verboten. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/offset/1 - ... using token - THEN HTTP status code is 400 Error - AND JSON body matches - ... { - ... "message": "LIMIT and OFFSET can only be used with together SORT.", - ... "error_code": "LimitWithoutSortError" - ... } - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/offset/1 - ... using token - THEN HTTP status code is 400 Error - AND JSON body matches - ... { - ... "message": "LIMIT and OFFSET can only be used with together SORT.", - ... "error_code": "LimitWithoutSortError" - ... } - - FINALLY qvarn is stopped - - -# Handle resource types via API - -Qvarn API allows listing and looking at all the resource types it -knows about. In the future, it will allow manipulaing them as well. - - SCENARIO manage resource types - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_resource_types_get uapi_resource_types_id_get" - - WHEN client requests GET /resource_types using token - THEN HTTP status code is 200 OK - AND search result contains { "id": "subject" } - - WHEN client requests GET /resource_types/subject using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "id": "subject", - ... "type": "resource_type", - ... "path": "/subjects" - ... } - - FINALLY qvarn is stopped |