Content¶
Parts of the module can be divided into two subcategories - scripts and parts of the API.
Scripts are meant to be used by users, API is there mainly for programmers.
Standalone scripts¶
Initializer script¶
This script is used to initialize ProFTPD and set configuration required by edeposit.amqp.ftp module.
It changes/creates ProFTPD configuration file, password file and extened log file. Also user directory is created and correct permissions is set.
Usage¶
$ ./edeposit_proftpd_init.py -h
usage: edeposit_proftpd_init.py [-h] [-o] [-v]
This script will modify your ProFTPD installation for use with
edeposit.amqp.ftp package.
optional arguments:
-h, --help show this help message and exit
-o, --overwrite Overwrite ProFTPD configuration file with edeposit.amqp.ftp
default configuration.
-v, --verbose Print debug output.
API¶
- edeposit_proftpd_init.main(*args, **kwargs)[source]¶
Used to create configuration files, set permissions and so on.
Monitor script¶
This script is used to monitor ProFTPD log and to react at certain events (deletion of the ftp.settings.LOCK_FILENAME).
It is also used at API level in edeposit.amqp (see process_log() and ftp_managerd).
Details of parsing are handled by request_parser.
- ftp.monitor._read_stdin()[source]¶
Generator for reading from standard input in nonblocking mode.
Other ways of reading from stdin in python waits, until the buffer is big enough, or until EOF character is sent.
This functions yields immediately after each line.
- ftp.monitor._parse_line(line)[source]¶
Convert one line from the extended log to dict.
Parameters: line (str) – Line which will be converted. Returns: dict with timestamp, command, username and path keys. Return type: dict Note
Typical line looks like this:
/home/ftp/xex/asd bsd.dat, xex, STOR, 1398351777
Filename may contain , character, so I am rsplitting the line from the end to the beginning.
- ftp.monitor.process_log(file_iterator)[source]¶
Process the extended ProFTPD log.
Parameters: file_iterator (file) – any file-like iterator for reading the log or stdin (see _read_stdin()). Yields: ImportRequest – with each import.
API¶
__init__.py¶
This module provides standard interface for AMQP communication as it is defined and used by edeposit.amqp.
The interface consists of reactToAMQPMessage() function, which receives two parameters - structure and UUID. UUID is not much important, but structure is usually namedtuple containing information what should module do.
After the work is done, reactToAMQPMessage() returns a value, which is then automatically transfered back to caller. If the exception is raised, it is also transfered in open and easy to handle way.
edeposit.amqp.ftp¶
In this module, reactToAMQPMessage() is used only for receiving commands from the other side. Events caused by FTP users are handled by monitor.py.
Commands can create/change/remove users and so on. This is done by sending one of the following structures defined in structures.py:
Responses¶
AddUser, RemoveUser and ChangePassword requests at this moment returns just simple True. This may be changed later.
ListRegisteredUsers returns Userlist class.
SetUserSettings and GetUserSettings both returns UserSettings structure.
API¶
- ftp.reactToAMQPMessage(message, send_back)[source]¶
React to given (AMQP) message. message is expected to be collections.namedtuple() structure from structures filled with all necessary data.
Parameters: - message (object) – One of the request objects defined in structures.
- send_back (fn reference) – Reference to function for responding. This is useful for progress monitoring for example. Function takes one parameter, which may be response structure/namedtuple, or string or whatever would be normally returned.
Returns: Response class from structures.
Return type: object
Raises: ValueError – if bad type of message structure is given.
Request parser¶
This submodule provides ability to process and parse import requests.
Most important function in this matter is the process_import_request(), which is called from from ftp.monitor.process_log(). When it is called, it scans the user’s home directory, detects new files, pairs them together into proper objects (see ftp.structures, speficifally MetadataFile, EbookFile and DataPair).
API¶
- ftp.request_parser.process_import_request(username, path, timestamp, logger_handler)[source]¶
React to import request. Look into user’s directory and react to files user uploaded there.
Behavior of this function can be set by setting variables in ftp.settings.
Parameters: Returns:
ProFTPD API¶
ProFTPD wrapped used to manage users of the FTP server.
This module controls the ftpd.passwd (LOGIN_FILE), creates/removes users directory and so on.
Warning
This API supposes, that it has permissions to read/write to ProFTPD configuration directory and to root directory for users.
Note
You don’t have to set the permissions and everything manually, there is script called initializer, which can do it for you automatically.
- ftp.api.reload_configuration(*args, **kwargs)[source]¶
Send signal to the proftpd daemon to reload configuration.
- ftp.api.recursive_chmod(path, mode=493)[source]¶
Recursively change mode for given path. Same as chmod -R mode.
Parameters: - path (str) – Path of the directory/file.
- mode (octal int, default 0755) – New mode of the file.
Warning
Don’t forget to add 0 at the beginning of the numbers of mode, or Unspeakable hOrRoRs will be awaken from their unholy sleep outside of the reality and they WILL eat your soul (and your files).
- ftp.api.create_lock_file(path)[source]¶
Create lock file filled with LOCK_FILE_CONTENT.
Parameters: path (str) – Path to the lock file. Made from users home directory and LOCK_FILENAME.
- ftp.api.add_user(*args, **kwargs)[source]¶
Adds record to passwd-like file for ProFTPD, creates home directory and sets permissions for important files.
Parameters:
- ftp.api.remove_user(*args, **kwargs)[source]¶
Remove user, his home directory and so on..
Parameters: username (str) – User’s name.
- ftp.api.list_users(*args, **kwargs)[source]¶
List all registered users, which are stored in LOGIN_FILE.
Returns: of str usernames. Return type: list
Passwd reader¶
API for reading/writing of the passwd file used by ProFTPD (and also unix).
API¶
- ftp.passwd_reader.load_users(path='/etc/proftpd/ftpd.passwd')[source]¶
Read passwd file and return dict with users and all their settings.
Parameters: path (str, default settings.LOGIN_FILE) – path of the file, which will be loaded (default ftp.settings.LOGIN_FILE). Returns: (dict): username: {pass_hash, uid, gid, full_name, home, shell} Example of returned data:
{ "xex": { "pass_hash": "$asd$aiosjdaiosjdásghwasdjo", "uid": "2000", "gid": "2000", "full_name": "ftftf", "home": "/home/ftp/xex", "shell": "/bin/false" } }
- ftp.passwd_reader.save_users(users, path='/etc/proftpd/ftpd.passwd')[source]¶
Save dictionary with user data to passwd file (default ftp.settings.LOGIN_FILE).
Parameters: - users (dict) – dictionary with user data. For details look at dict returned from load_users().
- path (str, default settings.LOGIN_FILE) – path of the file, where the data will be stored (default ftp.settings.LOGIN_FILE).
- ftp.passwd_reader.get_ftp_uid()[source]¶
Returns: UID of the proftpd/ftp user. Return type: int Raises: KeyError – When proftpd and ftp user is not found.
- ftp.passwd_reader.set_permissions(filename, uid=None, gid=None, mode=509)[source]¶
Set pemissions for given filename.
Parameters:
AMQP messages/structures¶
This module contains all communication structures used in AMQP communication.
Classes from Requests are used to manipulate FTP users.
Requests¶
User management requests¶
User requests¶
Settings management¶
- class ftp.structures.SetUserSettings[source]¶
Set settings for the user. UserSettings is returned as response.
See also
- CREATE_IMPORT_LOG¶
Alias for field number 4
- ISBN_PAIRING¶
Alias for field number 3
- LEAVE_BAD_FILES¶
Alias for field number 5
- SAME_DIR_PAIRING¶
Alias for field number 2
- SAME_NAME_DIR_PAIRING¶
Alias for field number 1
- username¶
Alias for field number 0
- class ftp.structures.GetUserSettings[source]¶
Get settings for given username.
UserSettings is returned as response.
See also
Responses¶
- class ftp.structures.Userlist[source]¶
Response containing names of all users.
- users¶
list
List of registered users.
- class ftp.structures.UserSettings[source]¶
All user settings, that user can set himself.
- CREATE_IMPORT_LOG¶
Alias for field number 4
- ISBN_PAIRING¶
Alias for field number 3
- LEAVE_BAD_FILES¶
Alias for field number 5
- SAME_DIR_PAIRING¶
Alias for field number 2
- SAME_NAME_DIR_PAIRING¶
Alias for field number 1
- username¶
Alias for field number 0
Import request¶
Import request are sent by monitor itself, without need of programmer interaction.
- class ftp.structures.ImportRequest[source]¶
User’s import request - mix of files, metadata and metadata-files pairs.
This request is sent asynchronously when user triggers the upload request.
- username¶
str
Name of the user who sent an import request.
- requests¶
list
List of MetadataFile/EbookFile/ DataPair objects.
- import_log¶
str
Protocol about import.
- error_log¶
str
Protocol about errors.
File structures¶
Following structures may be present in ImportRequest.requests.
- class ftp.structures.MetadataFile[source]¶
Structure used to represent Metadata files.
- filename¶
str
Name of the parsed file.
- raw_data¶
str
Content of the parsed file.
- parsed_data¶
EPublication
EPublication structure.
Decoders submodule¶
Decoders module used to parser metadata file into EPublication structure.
- ftp.decoders.parse_meta(filename, data)[source]¶
Parse data to EPublication.
Parameters: Returns: object.
Return type: EPublication
Available decoders¶
JSON decoder¶
This submodule is used to parse metadata from JSON (.json) files.
Metadata can be stored either in dictionary or in flat array.
Example structure:
{
"ISBN knihy": "80-86056-31-7",
"Vazba knihy": "brož.",
"Nazev knihy": "80-86056-31-7.json",
"Misto vydani": "Praha",
"Nakladatel": "Garda",
"Datum vydani": "09/2012",
"Poradi vydani": "1",
"Zpracovatel zaznamu": "Franta Putsalek"
}
or:
[
"ISBN knihy", "80-86056-31-7",
"Vazba knihy", "brož.",
"Nazev knihy", "samename.json",
"Misto vydani", "Praha",
"Nakladatel", "Garda",
"Datum vydani", "09/2012",
"Poradi vydani", "1",
"Zpracovatel zaznamu", "Franta Putsalek"
]
See Required fields for list of required fields.
XML decoder¶
This submodule is used to parse metadata from XML (.xml) files.
Format schema:
<root>
<item key="key">value</item>
</root>
Example of valid data:
<root>
<item key="ISBN knihy">80-86056-31-7</item>
<item key="Vazba knihy">brož.</item>
<item key="Nazev knihy">standalone2.xml</item>
<item key="Misto vydani">Praha</item>
<item key="Nakladatel">Garda</item>
<item key="Datum vydani">09/2012</item>
<item key="Poradi vydani">1</item>
<item key="Zpracovatel zaznamu">Franta Putsalek</item>
</root>
See Required fields for list of required fields.
CSV decoder¶
This submodule is used to parse metadata from CSV (.csv) files.
Example of the valid data:
ISBN knihy;978-80-87270-99-8
Vazba knihy;brož.
Nazev knihy;whatever.csv
Misto vydani;Praha
Nakladatel;Garda
Datum vydani;IX.12
Poradi vydani;1
Zpracovatel zaznamu;Franta Putsalek
See Required fields for list of required fields.
YAML decoder¶
This submodule is used to parse metadata from YAML (.yaml) files.
Example of the valid data:
ISBN knihy: 80-86056-31-7
Vazba knihy: brož.
Nazev knihy: 80-86056-31-7.json
Misto vydani: Praha
Nakladatel: Garda
Datum vydani: 09/2012
Poradi vydani: 1
Zpracovatel zaznamu: Franta Putsalek
See Required fields for list of required fields.
Other submodules¶
Validator¶
This module provides highlevel checking of parsed data for lowlevel decoders.
It handles the unicode in keys, builds dicts from flat arrays and so on.
- class ftp.decoders.validator.Field(keyword, descr, epub=None)[source]¶
This class is used to represent and parse specific “key: val” pair.
When you create the object, keyword and descr is specified. Optionally also epub parameter, which is corresponding key in EPublication structure.
Assingning value to the class is done by calling check(), which sets the value, if the key parameter matches keyword.
Parameters: - descr = None¶
Description of the data pair.
- epub = None¶
Corresponding key in EPublication structure.
- class ftp.decoders.validator.FieldParser[source]¶
Class used to make sure, that all fields in metadata are present.
See /api/required for list of required fields.
- process(key, val)[source]¶
Try to look for key in all required and optional fields. If found, set the val.
- get_epublication()[source]¶
Returns: Structure when the object is_valid(). Return type: EPublication Raises: MetaParsingException – When the object is not valid.
- ftp.decoders.validator.check_structure(data)[source]¶
Check whether the structure is flat dictionary. If not, try to convert it to dictionary.
Parameters: data – Whatever data you have (dict/tuple/list). Returns: When the conversion was successful or data was already good. Return type: dict Raises: MetaParsingException – When the data couldn’t be converted or had bad structure.
Settings and configuration¶
Module is containing all necessary global variables for the package.
Module also has the ability to read user-defined data from two paths:
- $HOME/_SETTINGS_PATH
- /etc/_SETTINGS_PATH
See _SETTINGS_PATH for details.
Note
If the first path is found, other is ignored.
Example of the configuration file ($HOME/edeposit/ftp.json):
{
"CONF_PATH": "/home/bystrousak/.ftpdconf/"
}
Attributes¶
- ftp.settings.BASE_PATH = '/var/build/user_builds/edeposit-amqp-ftp/checkouts/latest/src/edeposit/amqp/ftp'¶
Module’s path.
- ftp.settings.CONF_PATH = '/etc/proftpd/'¶
Proftpd configuration directory.
- ftp.settings.LOG_PATH = '/var/log/proftpd/'¶
Proftpd log directory.
- ftp.settings.DATA_PATH = '/home/ftp/'¶
Path to directory, where the user directories will be created.
- ftp.settings.SERVER_ADDRESS = 'localhost'¶
Server’s address - used only in unit/integration testing.
- ftp.settings.CONF_FILE = '/etc/proftpd/proftpd.conf'¶
Proftpd configuration file (in CONF_PATH directory).
- ftp.settings.LOGIN_FILE = '/etc/proftpd/ftpd.passwd'¶
File where the login informations will be stored (CONF_PATH is used as dirname).
- ftp.settings.LOG_FILE = '/var/log/proftpd/extended.log'¶
File where the extended logs are stored (LOG_PATH is used as dirname).
- ftp.settings.LOCK_FILENAME = 'delete_me_to_import_files.txt'¶
Filename for the locking mechanism.
- ftp.settings.USER_ERROR_LOG = 'error.log.txt'¶
Filename, where the error protocol is stored.
- ftp.settings.USER_IMPORT_LOG = 'import.log.txt'¶
Filename, where the import protocol for the user is stored.
- ftp.settings.LOCK_FILE_CONTENT = "Delete this file to start batch import of all files, you've uploaded to the server.\n\nSmazte tento soubor pro zapoceti davkoveho importu vsech souboru, ktere jste\nnahrali na server.\n"¶
Text, which will be writen to the PROTFPD_LOCK_FILENAME.
- ftp.settings.SAME_NAME_DIR_PAIRING = True¶
True - will pair files with same filename in same directory
- ftp.settings.SAME_DIR_PAIRING = True¶
True - will pair files with different filenames, if there is only two files in dir
- ftp.settings.ISBN_PAIRING = True¶
True - if the name is ISBN, files will be paired no matter where they are stored (unless they weren’t paired before)
- ftp.settings.LOCK_ONLY_IN_HOME = True¶
True - Lock file can be only in home directory, everywhere else will be ignored
- ftp.settings.CREATE_IMPORT_LOG = True¶
True - USER_IMPORT_LOG will be created
- ftp.settings.LEAVE_BAD_FILES = True¶
True - don’t remove badly formatted metadata files
- ftp.settings.PROFTPD_USERS_GID = 2000¶
I am using GID 2000 for all our users - this GID shouldn’t be used by other than FTP users!
- ftp.settings.conf_merger(user_dict, variable)[source]¶
Merge global configuration with user’s personal configuration.
Global configuration has always higher priority.
- ftp.settings.get_all_constants()[source]¶
Get list of all uppercase, non-private globals (doesn’t start with _).
Returns: Uppercase names defined in globals() (variables from this module). Return type: list
- ftp.settings.substitute_globals(config_dict)[source]¶
Set global variables to values defined in config_dict.
Parameters: config_dict (dict) – dictionary with data, which are used to set globals. Note
config_dict have to be dictionary, or it is ignored. Also all variables, that are not already in globals, or are not types defined in _ALLOWED (str, int, float) or starts with _ are silently ignored.