In-Portal Issue Tracker

Welcome to the In-Portal Open Source CMS Issue Tracker! This is a central management / tracking tool for all types of tasks / issues / bugs for the In-Portal Project. Before reporting any issues, please make sure to read the Guide into Issue Tracker and How to Properly Test and Report Bugs!

Viewing Issue Advanced Details Jump to Notes ] Wiki ]  Related Changesets ] View Simple ] Issue History ] Print ]
ID Category Type Reproducibility Date Submitted Last Update
0001022 [In-Portal CMS] Data Management feature request N/A 2011-03-13 07:21 2012-07-25 05:32
Reporter alex View Status public Project Name In-Portal CMS
Assigned To alex Developer
Priority normal Resolution fixed Platform
Status closed   OS
  OS Version
ETA none Fixed in Version 5.2.0-B1 Product Version 5.1.2-B2
  Target Version 5.2.0 Product Build
Time EstimateNo estimate
Summary 0001022: Uploaded file distribution within /system/ folder
Description When file uploads are happening a lot on the site, then all files are stored under same /system/... sub-folder. This doesn't pose any problem to webserver to easily access them, however it makes it near to impossible to manage that files manually. Like view/backup/restore individual files, when there are 2000+ files in one directory.

That's why we are proposing uploaded file distribution engine, that will create sub-folders within target upload folder to store uploaded files. Here are some ideas:

1. Filenames

a. ability to have Prefix (defined in Field declaration (in unit config), default is empty, DATE-TIME which reserved, but can specify any other symbols)

b. ability to have Ending (defined in Field declaration (in unit config), default is DATE-TIME which reserved, but can specify any other symbols)

DATE = 20110131 (2011/01/31)
TIME = 123401 (12:34:01)

Examples:

- 20110131-123401_my-custom-filename-whatever.jpg
- my-custom-filename-whatever_20110131-123401.jpg

or other combinations

This will give us full flexibility in filenames


2. Folders

Similar here, give 3 options for each upload field:

a. As is now - All files go into single specified folder.

b. Chronological - YEARMONTH / DAY / (adds 2 sub-levels)

c. Hash - as proposed by Alex (adds 2 sub-levels, see "Additional Information" field for function code)


NOTE:
1. Hashed path will be stored along with a filename, since in case of timestamp-based hashing there is no way to determine hash path to a file by it's name.
2. All resized versions will be placed without special structure under /resized/ folder.
3. All code completely/mostly will be located in kUploadFormatter class, so extending that class could give ability to add new storage engine types
Steps To Reproduce
Additional Information Hashing function for 2c item:

static function getHashPathForLevel( $name, $levels = 2 ) {
    if ( $levels == 0 ) {
        return '';
    } else {
        $hash = md5( $name );
        $path = '';
        for ( $i = 1; $i <= $levels; $i++ ) {
            $path .= substr( $hash, 0, $i ) . '/';
        }
        return $path;
    }
}


Unit config field declaration:
'DataFile' => Array (
    'type' => 'string', 'formatter' => 'kUploadFormatter',
    'max_size' => MAX_UPLOAD_SIZE, // in Bytes !
    'file_types' => '*.*',
    'files_description' => '!la_hint_AllFiles!',
    'upload_dir' => '/system/user_files/', // relative to project's home
    'as_image' => false,
    'multiple' => 5, // false or max number of files - will be stored as serialized array of paths
    'direct_links' => true, // use direct file urls or send files through wrapper (requires mod_mime_magic)
    'default' => null,
    'filename_prefix' => 'DATE-TIME-PREFIX-USER', // alpha-numeric symbol or phrase, or reserved words (DATE/TIME/PREFIX/USER) that will be added as a Prefix to filename
    'filename_suffix' => 'DATE-TIME-PREFIX-USER', // alpha-numeric symbol or phrase, or reserved words (DATE/TIME/PREFIX/USER) that will be added as an Ending to filename
    'storage_engine' => STORAGE_ENGINE_NONE (default) or STORAGE_ENGINE_HASH or STORAGE_ENGINE_TIMESTAMP , // option to specify desired storing engine: _blank_, HASH (2 level sub-folder generated from filename hash), TIMESTAMP (2 level sub-folders as yearmonth/day)
)
Tags No tags attached.
Reference https://groups.google.com/d/topic/in-portal-dev/1MoriT0tjE8/discussion
Change Log Message Adding file distributed storage engine
Estimate Points 3
Attached Files patch file icon upload_file_distribution_1022.patch [^] (6,079 bytes) 2011-12-13 05:57 [Show Content]
patch file icon upload_file_distribution_core_v2.patch [^] (13,456 bytes) 2011-12-15 08:38 [Show Content]
patch file icon upload_file_distribution_modules_v2.patch [^] (693 bytes) 2011-12-15 08:39 [Show Content]

- Relationships Relation Graph ] Dependency Graph ]

-  Notes
User avatar (0004256)
erik (manager)
2011-12-13 05:58

Patch uploaded - needs testing
User avatar (0004262)
alex (manager)
2011-12-15 08:39

Fixed problems:
1. when storage engine used in "images_config.php" file, then images were broken after saving from temp table happened
2. when prefix/suffix plus something else was specified in unit config (e.g. "USER_test"), then "USER" (or any other special keyword) wasn't substituted
3. new method were made "private" resulting inability to override them in extended classes
4. new method PHPDoc comments were not matching actual method declaration
5. spelling errors found
6. storage engine sub-folder wasn't included in unique filename check when non-flash uploader was used

Changes:
1. changed "Ym/d" (e.g. "201112/15") folder format to "Y-m/d" (e.g. "2011-12/15")
2. added "u" in "USER" keyword result (was "_15", now "_u15") to prevent accidental removal by FileHelper::ensureUniqueFilename method
3. exception is raised in case if unsupported storage engine name is specified
4. changed new variable/method namings to fix spelling errors in them
5. added sample code in "widgets_config.php" (otherwise no way for developers to know that we've added something new)
6. constants converted to class format
7. FileHelper::ensureUniqueFilename now also creates folder, that is used to perform check
8. FileHelper::ensureUniqueFilename now also allows a storage folder to be supplied as filename part (will be returned back in result filename too)
9. FileHelper::ensureUniqueFilename changed to use PHP 5.2 version of "pathinfo" function
10. ImageEventHandler::ImageAction method now uses FileHelper::ensureUniqueFilename method instead of it's ValidateFileName outdated copy
User avatar (0004263)
alex (manager)
2011-12-15 08:40

Fix committed to 5.2.x branch. Commit Message:

Fixes 0001022: Uploaded file distribution within /system/ folder
User avatar (0004264)
alex (manager)
2011-12-15 08:54

It's not possible to keep all resized images in top folder (where they were before) because:

1. full image path is given to ImageHelper::ResizeImage method
2. no way to guess what part of the path is created by storage engine and if it's present at all
User avatar (0005024)
alex (manager)
2012-07-25 05:32

Since 5.2.0 version was released.

- Related Changesets
In-Portal CMS: 5.2.x r14868
Timestamp: 2011-12-15 08:40:59
Author: alex
Details ] Diff ]
Fixes 0001022: Uploaded file distribution within /system/ folder
mod - /in-portal/branches/5.2.x/core/kernel/constants.php Diff ] File ]
mod - /in-portal/branches/5.2.x/core/kernel/globals.php Diff ] File ]
mod - /in-portal/branches/5.2.x/core/kernel/utility/formatters/upload_formatter.php Diff ] File ]
mod - /in-portal/branches/5.2.x/core/units/helpers/file_helper.php Diff ] File ]
mod - /in-portal/branches/5.2.x/core/units/images/image_event_handler.php Diff ] File ]
Modules :: Custom: 1.2.x r14867
Timestamp: 2011-12-15 08:40:24
Author: alex
Details ] Diff ]
Bug 0001022: Uploaded file distribution within /system/ folder
mod - /modules/custom/branches/1.2.x/units/widgets/widgets_config.php Diff ] File ]

- Issue History
Date Modified Username Field Change
2012-07-25 05:32 alex Note Added: 0005024
2012-07-25 05:32 alex Status resolved => closed
2011-12-15 08:54 alex Note Added: 0004264
2011-12-15 08:40 alex Note Added: 0004263
2011-12-15 08:40 alex Status reviewed and tested => resolved
2011-12-15 08:40 alex Fixed in Version => 5.2.0-B1
2011-12-15 08:40 alex Resolution open => fixed
2011-12-15 08:40 alex Assigned To !COMMUNITY => alex
2011-12-15 08:40 alex Changeset attached 5.2.x r14868
2011-12-15 08:40 alex Changeset attached 1.2.x r14867
2011-12-15 08:40 alex Target Version Icebox => 5.2.0
2011-12-15 08:39 alex Note Added: 0004262
2011-12-15 08:39 alex Assigned To alex => !COMMUNITY
2011-12-15 08:39 alex Status needs testing => reviewed and tested
2011-12-15 08:39 alex File Added: upload_file_distribution_modules_v2.patch
2011-12-15 08:38 alex File Added: upload_file_distribution_core_v2.patch
2011-12-15 02:53 alex Estimate Points 0 => 3
2011-12-13 05:58 erik Note Added: 0004256
2011-12-13 05:58 erik Assigned To erik => alex
2011-12-13 05:58 erik Developer => erik
2011-12-13 05:58 erik Status needs work => needs testing
2011-12-13 05:57 erik File Added: upload_file_distribution_1022.patch
2011-12-12 08:17 Dmitry Assigned To => erik
2011-12-12 08:17 Dmitry Status active => needs work
2011-11-06 20:39 Dmitry Reference => https://groups.google.com/d/topic/in-portal-dev/1MoriT0tjE8/discussion
2011-11-06 20:39 Dmitry Estimate Points => 0
2011-03-13 07:21 alex New Issue
2011-03-13 07:21 alex Change Log Message => Adding file distributed storage engine



Web Development by Intechnic
In-Portal Open Source CMS
In-Portal Open Source CMS
Copyright © 2000 - 2009 MantisBT Group

Powered by Mantis Bugtracker