0001022: Uploaded file distribution within /system/ folder

1022

[In-Portal CMS] Data Management

feature request

N/A

2011-03-13 07:21

2012-07-25 05:32

Reporter:

alex

Platform:

Assigned To:

alex

OS:

Priority:

normal

OS Version:

Status:

closed

Product Version:

5.1.2-B2

Product Build:

Resolution:

fixed

ETA:

none

Fixed in Version:

5.2.0-B1

Reference:

https://groups.google.com/d/topic/in-portal-dev/1MoriT0tjE8/discussion

Change Log Message:

Adding file distributed storage engine

Estimate Points:

3

Summary:

0001022: Uploaded file distribution within /system/ folder

Description:

When file uploads are happening a lot on the site, then all files are stored under same /system/... sub-folder. This doesn't pose any problem to webserver to easily access them, however it makes it near to impossible to manage that files manually. Like view/backup/restore individual files, when there are 2000+ files in one directory.

That's why we are proposing uploaded file distribution engine, that will create sub-folders within target upload folder to store uploaded files. Here are some ideas:

1. Filenames

a. ability to have Prefix (defined in Field declaration (in unit config), default is empty, DATE-TIME which reserved, but can specify any other symbols)

b. ability to have Ending (defined in Field declaration (in unit config), default is DATE-TIME which reserved, but can specify any other symbols)

DATE = 20110131 (2011/01/31)
TIME = 123401 (12:34:01)

Examples:

- 20110131-123401_my-custom-filename-whatever.jpg
- my-custom-filename-whatever_20110131-123401.jpg

or other combinations

This will give us full flexibility in filenames

2. Folders

Similar here, give 3 options for each upload field:

a. As is now - All files go into single specified folder.

b. Chronological - YEARMONTH / DAY / (adds 2 sub-levels)

c. Hash - as proposed by Alex (adds 2 sub-levels, see "Additional Information" field for function code)

NOTE:
1. Hashed path will be stored along with a filename, since in case of timestamp-based hashing there is no way to determine hash path to a file by it's name.
2. All resized versions will be placed without special structure under /resized/ folder.
3. All code completely/mostly will be located in kUploadFormatter class, so extending that class could give ability to add new storage engine types

Steps To Reproduce:

Additional Information:

Hashing function for 2c item:

static function getHashPathForLevel( $name, $levels = 2 ) {
    if ( $levels == 0 ) {
        return '';
    } else {
        $hash = md5( $name );
        $path = '';
        for ( $i = 1; $i <= $levels; $i++ ) {
            $path .= substr( $hash, 0, $i ) . '/';
        }
        return $path;
    }
}

Unit config field declaration:
'DataFile' => Array (
    'type' => 'string', 'formatter' => 'kUploadFormatter',
    'max_size' => MAX_UPLOAD_SIZE, // in Bytes !
    'file_types' => '*.*',
    'files_description' => '!la_hint_AllFiles!',
    'upload_dir' => '/system/user_files/', // relative to project's home
    'as_image' => false,
    'multiple' => 5, // false or max number of files - will be stored as serialized array of paths
    'direct_links' => true, // use direct file urls or send files through wrapper (requires mod_mime_magic)
    'default' => null,
    'filename_prefix' => 'DATE-TIME-PREFIX-USER', // alpha-numeric symbol or phrase, or reserved words (DATE/TIME/PREFIX/USER) that will be added as a Prefix to filename
    'filename_suffix' => 'DATE-TIME-PREFIX-USER', // alpha-numeric symbol or phrase, or reserved words (DATE/TIME/PREFIX/USER) that will be added as an Ending to filename
    'storage_engine' => STORAGE_ENGINE_NONE (default) or STORAGE_ENGINE_HASH or STORAGE_ENGINE_TIMESTAMP , // option to specify desired storing engine: _blank_, HASH (2 level sub-folder generated from filename hash), TIMESTAMP (2 level sub-folders as yearmonth/day)
)

Relationships

Attached Files:

upload_file_distribution_1022.patch (6,079) 2011-12-13 05:57
http://tracker.in-portal.org/file_download.php?file_id=1373&type=bug

upload_file_distribution_core_v2.patch (13,456) 2011-12-15 08:38
http://tracker.in-portal.org/file_download.php?file_id=1377&type=bug

upload_file_distribution_modules_v2.patch (693) 2011-12-15 08:39
http://tracker.in-portal.org/file_download.php?file_id=1378&type=bug

2012-07-25 05:32

In-Portal Issue Tracker - In-Portal CMS
Viewing Issue Advanced Details

ID:	Category:	Type:	Reproducibility:	Date Submitted:	Last Update:
1022	[In-Portal CMS] Data Management	feature request	N/A	2011-03-13 07:21	2012-07-25 05:32

Reporter:	alex	Platform:
Assigned To:	alex	OS:
Priority:	normal	OS Version:
Status:	closed	Product Version:	5.1.2-B2
Product Build:		Resolution:	fixed

ETA:	none	Fixed in Version:	5.2.0-B1
Reference:	https://groups.google.com/d/topic/in-portal-dev/1MoriT0tjE8/discussion
Change Log Message:	Adding file distributed storage engine
Estimate Points:	3

Summary:	0001022: Uploaded file distribution within /system/ folder
Description:	When file uploads are happening a lot on the site, then all files are stored under same /system/... sub-folder. This doesn't pose any problem to webserver to easily access them, however it makes it near to impossible to manage that files manually. Like view/backup/restore individual files, when there are 2000+ files in one directory. That's why we are proposing uploaded file distribution engine, that will create sub-folders within target upload folder to store uploaded files. Here are some ideas: 1. Filenames a. ability to have Prefix (defined in Field declaration (in unit config), default is empty, DATE-TIME which reserved, but can specify any other symbols) b. ability to have Ending (defined in Field declaration (in unit config), default is DATE-TIME which reserved, but can specify any other symbols) DATE = 20110131 (2011/01/31) TIME = 123401 (12:34:01) Examples: - 20110131-123401_my-custom-filename-whatever.jpg - my-custom-filename-whatever_20110131-123401.jpg or other combinations This will give us full flexibility in filenames 2. Folders Similar here, give 3 options for each upload field: a. As is now - All files go into single specified folder. b. Chronological - YEARMONTH / DAY / (adds 2 sub-levels) c. Hash - as proposed by Alex (adds 2 sub-levels, see "Additional Information" field for function code) NOTE: 1. Hashed path will be stored along with a filename, since in case of timestamp-based hashing there is no way to determine hash path to a file by it's name. 2. All resized versions will be placed without special structure under /resized/ folder. 3. All code completely/mostly will be located in kUploadFormatter class, so extending that class could give ability to add new storage engine types
Steps To Reproduce:
Additional Information:	Hashing function for 2c item: static function getHashPathForLevel( $name, $levels = 2 ) { if ( $levels == 0 ) { return ''; } else { $hash = md5( $name ); $path = ''; for ( $i = 1; $i <= $levels; $i++ ) { $path .= substr( $hash, 0, $i ) . '/'; } return $path; } } Unit config field declaration: 'DataFile' => Array ( 'type' => 'string', 'formatter' => 'kUploadFormatter', 'max_size' => MAX_UPLOAD_SIZE, // in Bytes ! 'file_types' => '.', 'files_description' => '!la_hint_AllFiles!', 'upload_dir' => '/system/user_files/', // relative to project's home 'as_image' => false, 'multiple' => 5, // false or max number of files - will be stored as serialized array of paths 'direct_links' => true, // use direct file urls or send files through wrapper (requires mod_mime_magic) 'default' => null, 'filename_prefix' => 'DATE-TIME-PREFIX-USER', // alpha-numeric symbol or phrase, or reserved words (DATE/TIME/PREFIX/USER) that will be added as a Prefix to filename 'filename_suffix' => 'DATE-TIME-PREFIX-USER', // alpha-numeric symbol or phrase, or reserved words (DATE/TIME/PREFIX/USER) that will be added as an Ending to filename 'storage_engine' => STORAGE_ENGINE_NONE (default) or STORAGE_ENGINE_HASH or STORAGE_ENGINE_TIMESTAMP , // option to specify desired storing engine: _blank_, HASH (2 level sub-folder generated from filename hash), TIMESTAMP (2 level sub-folders as yearmonth/day) )
Relationships
Attached Files:	upload_file_distribution_1022.patch (6,079) 2011-12-13 05:57 http://tracker.in-portal.org/file_download.php?file_id=1373&type=bug upload_file_distribution_core_v2.patch (13,456) 2011-12-15 08:38 http://tracker.in-portal.org/file_download.php?file_id=1377&type=bug upload_file_distribution_modules_v2.patch (693) 2011-12-15 08:39 http://tracker.in-portal.org/file_download.php?file_id=1378&type=bug

Issue History
Date Modified	Username	Field	Change
2012-07-25 05:32	alex	Note Added: 0005024
2012-07-25 05:32	alex	Status	resolved => closed
2011-12-15 08:54	alex	Note Added: 0004264
2011-12-15 08:40	alex	Note Added: 0004263
2011-12-15 08:40	alex	Status	reviewed and tested => resolved
2011-12-15 08:40	alex	Fixed in Version	=> 5.2.0-B1
2011-12-15 08:40	alex	Resolution	open => fixed
2011-12-15 08:40	alex	Assigned To	!COMMUNITY => alex
2011-12-15 08:40	alex	Changeset attached	5.2.x r14868
2011-12-15 08:40	alex	Changeset attached	1.2.x r14867
2011-12-15 08:40	alex	Target Version	Icebox => 5.2.0
2011-12-15 08:39	alex	Note Added: 0004262
2011-12-15 08:39	alex	Assigned To	alex => !COMMUNITY
2011-12-15 08:39	alex	Status	needs testing => reviewed and tested
2011-12-15 08:39	alex	File Added: upload_file_distribution_modules_v2.patch
2011-12-15 08:38	alex	File Added: upload_file_distribution_core_v2.patch
2011-12-15 02:53	alex	Estimate Points	0 => 3
2011-12-13 05:58	erik	Note Added: 0004256
2011-12-13 05:58	erik	Assigned To	erik => alex
2011-12-13 05:58	erik	Developer	=> erik
2011-12-13 05:58	erik	Status	needs work => needs testing
2011-12-13 05:57	erik	File Added: upload_file_distribution_1022.patch
2011-12-12 08:17	Dmitry	Assigned To	=> erik
2011-12-12 08:17	Dmitry	Status	active => needs work
2011-11-06 20:39	Dmitry	Reference	=> https://groups.google.com/d/topic/in-portal-dev/1MoriT0tjE8/discussion
2011-11-06 20:39	Dmitry	Estimate Points	=> 0
2011-03-13 07:21	alex	New Issue
2011-03-13 07:21	alex	Change Log Message	=> Adding file distributed storage engine