In-Portal Issue Tracker - In-Portal CMS
Viewing Issue Advanced Details
ID: Category: Type: Reproducibility: Date Submitted: Last Update:
107 [In-Portal CMS] Optimization feature request always 2009-07-07 09:53 2012-08-11 12:20
Reporter: Dmitry Platform:  
Assigned To: alex OS:  
Priority: normal OS Version:  
Status: closed Product Version:  
Product Build: Resolution: fixed  
 
ETA: none Fixed in Version: 5.1.0-B1  
Reference:
Change Log Message:
Estimate Points: 0
Summary: 0000107: Implement "MemCached" functionality
Description: Add MemCached implementation to NParser.

1. Caching for:

- ConfigReaders
- ModRewrite URL variables (serialized)
- Permissions (double-check on Groups, possibly GroupId is not in Key now, overall Key in Memcache should contain all parameters that are used for getting given permission value).
- Output (by CategoryRebuildSerial, template/std, CategoryId)
- Other possible places


2. Review and add good Reset Cache options (full reset or partial)


3. Configuration of MemCached servers - should specified in Configuration->Advanced Settings.


4. Review Security options.
Steps To Reproduce:
Additional Information: We can utilize VO project as it has basic MemCached implemented and used already.
Relationships
related to 0000049closed  (5.1.0)alex Merge ConfigurationAdmin and ConfigurationValues table 
related to 0000588closed  (5.1.0)alex Implement "APC" Caching Storage 
related to 0001326closed  (5.2.0)alex Fatal error when trying to use parser tag caching in 5.2.0 on PHP version below 5.3.1 
parent of 0000357closed  (5.1.0)alex Form modification detector (it's php part) works strange 
parent of 0000884closed  (5.1.1)alex Fatal error "Method name must be a string" on home page after Memcached server restart 
parent of 0000885closed  (5.1.1)alex Additional images of category items are not displayed (with memory caching enabled) 
parent of 0000958closed  (5.1.2)alex Editing language on section edit form don't change on primary language change 
related to 0000579closed  (5.1.0)alex Ability to use PerPage, SortBy and Page params from URL Query 
related to 0000580closed  (5.1.1)Dmitry Issues with change log section 
related to 0000812closed  (5.1.1)alex Sticky cache, when Memcached is not used 
related to 0000882closed  (5.1.1)alex Memcache issues 
related to 0000877closed  (5.1.1)alex Single Cache Serial-key for Developers on the same Server 
related to 0001028closed  (5.1.3)alex Missing "Database Name" in Memory Cache Key 
related to 0001123closed  (5.2.0)alex Parallel cache rebuild problem could cause high server load 
related to 0001201closed  (5.2.0)alex Fatal error during Installation on States import with MySQL in Strict Mode 
related to 0001332closed  (5.2.0)alex Spelling suggestion cache not reset on dictionary change 
child of 0000296needs work  (Icebox)alex Research and Create Optimization Plan 
Attached Files: vsd Caching_process.vsd (64,512) 2010-02-13 07:50
http://tracker.in-portal.org/file_download.php?file_id=289&type=bug
patch memcache_modules.patch (15,971) 2010-02-24 07:49
http://tracker.in-portal.org/file_download.php?file_id=301&type=bug
patch memcache_core.patch (172,247) 2010-02-24 10:46
http://tracker.in-portal.org/file_download.php?file_id=304&type=bug
patch memcache_themes.patch (24,485) 2010-02-24 12:04
http://tracker.in-portal.org/file_download.php?file_id=305&type=bug
patch memcache_notification_addon.patch (867) 2010-03-02 14:18
http://tracker.in-portal.org/file_download.php?file_id=319&type=bug
patch memcache_theme_rebuild_fix.patch (863) 2010-03-07 13:29
http://tracker.in-portal.org/file_download.php?file_id=327&type=bug
patch memcache_install_fix.patch (478) 2010-03-21 16:29
http://tracker.in-portal.org/file_download.php?file_id=391&type=bug
patch memcache_install_fix_v2.patch (1,035) 2010-03-24 08:33
http://tracker.in-portal.org/file_download.php?file_id=402&type=bug
patch memcache_custom_field_caching_on_install.patch (2,078) 2010-03-24 08:39
http://tracker.in-portal.org/file_download.php?file_id=403&type=bug
patch cached_permissions_not_updated_after_group_edit.patch (1,651) 2010-04-13 12:40
http://tracker.in-portal.org/file_download.php?file_id=464&type=bug
jpg QuestionAboutCachingNumberOfProductsInCategory.jpg (580,125) 2010-04-25 18:24
http://tracker.in-portal.org/file_download.php?file_id=481&type=bug
patch overlaped_cache_names_fix.patch (477) 2010-05-12 07:28
http://tracker.in-portal.org/file_download.php?file_id=539&type=bug
patch memcache_addon.patch (6,083) 2010-05-24 14:50
http://tracker.in-portal.org/file_download.php?file_id=592&type=bug
Issue History
Date Modified Username Field Change
2012-08-11 12:20 Dmitry Relationship added related to 0001326
2012-06-28 05:00 alex Relationship added related to 0001332
2012-02-06 04:09 alex Relationship added related to 0001201
2011-09-19 15:59 alex Relationship added related to 0001123
2011-03-31 11:06 Dmitry Relationship added related to 0001028
2011-01-04 05:43 alex Relationship added parent of 0000958
2010-10-11 12:46 alex Relationship added parent of 0000885
2010-10-11 11:12 alex Relationship added related to 0000877
2010-10-11 11:12 alex Relationship added related to 0000882
2010-10-10 10:19 alex Relationship added parent of 0000884
2010-07-27 14:03 alex Relationship added related to 0000812
2010-07-22 15:06 alex Note Added: 0002521
2010-07-22 15:06 alex Status resolved => closed
2010-05-24 15:05 alex Changeset attached 5.1.x r13614
2010-05-24 15:05 alex Note Added: 0002314
2010-05-24 15:05 alex Status needs work => resolved
2010-05-24 15:05 alex Fixed in Version => 5.1.0-B1
2010-05-24 15:05 alex Resolution reopened => fixed
2010-05-24 15:03 alex Note Added: 0002313
2010-05-24 14:50 alex File Added: memcache_addon.patch
2010-05-23 19:48 Dmitry Note Added: 0002301
2010-05-12 07:31 alex Changeset attached 5.1.x r13561
2010-05-12 07:28 alex File Added: overlaped_cache_names_fix.patch
2010-04-26 06:38 Dmitry Note Added: 0001955
2010-04-26 06:38 Dmitry Assigned To Dmitry => alex
2010-04-26 06:38 Dmitry Status needs feedback => needs work
2010-04-26 04:59 alex Assigned To alex => Dmitry
2010-04-26 04:59 alex Note Added: 0001951
2010-04-25 18:26 Dmitry Note Edited: 0001938 bug_revision_view_page.php?bugnote_id=0001938#r449
2010-04-25 18:26 Dmitry Note Edited: 0001938 bug_revision_view_page.php?bugnote_id=0001938#r448
2010-04-25 18:24 Dmitry Note Edited: 0001938 bug_revision_view_page.php?bugnote_id=0001938#r447
2010-04-25 18:24 Dmitry File Added: QuestionAboutCachingNumberOfProductsInCategory.jpg
2010-04-25 18:23 Dmitry Note Added: 0001938
2010-04-25 18:23 Dmitry Status resolved => needs feedback
2010-04-25 18:23 Dmitry Resolution fixed => reopened
2010-04-13 12:45 alex Changeset attached 5.1.x r13382
2010-04-13 12:41 alex Note Added: 0001880
2010-04-13 12:40 alex File Added: cached_permissions_not_updated_after_group_edit.patch
2010-03-24 08:41 alex Changeset attached 5.1.x r13313
2010-03-24 08:40 alex Note Added: 0001742
2010-03-24 08:39 alex File Added: memcache_custom_field_caching_on_install.patch
2010-03-24 08:36 alex Changeset attached 5.1.x r13312
2010-03-24 08:35 alex Note Added: 0001741
2010-03-24 08:33 alex File Added: memcache_install_fix_v2.patch
2010-03-21 16:32 alex Changeset attached 5.1.x r13275
2010-03-21 16:31 alex Note Added: 0001719
2010-03-21 16:29 alex File Added: memcache_install_fix.patch
2010-03-08 01:22 alex Note Added: 0001522
2010-03-07 14:47 Dmitry Note Added: 0001521
2010-03-07 13:33 alex Changeset attached 5.1.x r13182
2010-03-07 13:32 alex Note Added: 0001520
2010-03-07 13:29 alex File Added: memcache_theme_rebuild_fix.patch
2010-03-02 14:21 alex Changeset attached 5.1.x r13173
2010-03-02 14:20 alex Note Added: 0001508
2010-03-02 14:18 alex File Added: memcache_notification_addon.patch
2010-03-01 14:20 Dmitry Relationship added related to 0000588
2010-02-27 05:00 alex Developer => alex
2010-02-24 12:06 alex Note Added: 0001489
2010-02-24 12:04 alex Changeset attached 1.1.x r13169
2010-02-24 12:04 alex Note Added: 0001488
2010-02-24 12:04 alex Fixed in Version 5.1.0-B1 =>
2010-02-24 12:04 alex File Added: memcache_themes.patch
2010-02-24 11:56 alex Time Spent Added 50 hours
2010-02-24 11:55 alex Relationship added related to 0000049
2010-02-24 11:28 alex Note Added: 0001486
2010-02-24 11:26 alex Note Added: 0001485
2010-02-24 11:26 alex Status needs work => resolved
2010-02-24 11:26 alex Fixed in Version => 5.1.0-B1
2010-02-24 11:26 alex Resolution open => fixed
2010-02-24 11:26 alex Changeset attached 5.1.x r13168
2010-02-24 10:46 alex File Added: memcache_core.patch
2010-02-24 08:14 alex Changeset attached 5.1.x r13165
2010-02-24 08:13 alex Changeset attached 5.1.x r13164
2010-02-24 07:51 alex Changeset attached 5.1.x r13163
2010-02-24 07:49 alex File Added: memcache_modules.patch
2010-02-24 07:29 alex Relationship added related to 0000580
2010-02-16 17:22 Dmitry Note Added: 0001480
2010-02-15 16:22 Dmitry Relationship added related to 0000579
2010-02-15 16:18 Dmitry Note Added: 0001479
2010-02-13 07:50 alex File Added: Caching_process.vsd
2010-02-11 14:35 Dmitry Status needs feedback => needs work
2010-02-11 14:34 Dmitry Relationship added child of 0000296
2010-02-11 14:34 alex Relationship added parent of 0000357
2010-01-12 12:12 alex Target Version Icebox => 5.1.0
2010-01-12 11:34 alex Target Version 5.1.0 => Icebox
2009-10-03 07:47 administrator Priority @50@ => normal
2009-09-15 01:18 Dmitry Priority critical => @50@
2009-09-15 01:11 Dmitry Priority normal => critical
2009-09-15 01:11 Dmitry Note Added: 0000441
2009-09-15 01:11 Dmitry Assigned To Dmitry => alex
2009-09-15 01:11 Dmitry Status reviewed and tested => needs feedback
2009-09-14 09:45 Dmitry Note Added: 0000434
2009-09-14 09:45 Dmitry Target Version 5.2.x => 5.1.0
2009-09-13 05:29 alex Note Added: 0000423
2009-07-07 10:03 Dmitry Additional Information Updated bug_revision_view_page.php?rev_id=92#r92
2009-07-07 10:02 Dmitry Description Updated bug_revision_view_page.php?rev_id=90#r90
2009-07-07 09:57 Dmitry Description Updated bug_revision_view_page.php?rev_id=89#r89
2009-07-07 09:53 Dmitry New Issue
2009-07-07 09:53 Dmitry Status active => reviewed and tested
2009-07-07 09:53 Dmitry Assigned To => Dmitry

Notes
(0000423)
alex   
2009-09-13 05:29   
Also we could think up a way to cache other parts of system, like results of database queries for "options_sql" option of kOptionsFormatter class.
(0000434)
Dmitry   
2009-09-14 09:45   
Moving this to Optimization 5.1.0 release.
(0000441)
Dmitry   
2009-09-15 01:11   
1. Do complete review of In-Portal system - do detailed review using profiler.

2. Come up with architecture so we can Cache:

a. Permissions
b. SQLs
c. Config Data
d. Mod-rewrite variables
e. Parts of the Output
f. Entire templates
g. Anything else you can think of?

3. We should be able to Specify Caching time (I know MemCache doesn't have proper way of resetting the Cache) + Ability to reset it manually for each specified or all variables.

In the end we need detailed plan / quote for development.
(0001479)
Dmitry   
2010-02-15 16:18   
Below is a plan for Aggressive Caching part:

1. collect prefixes from template (associated with page)

2. put them into Category (or ThemeFiles) table

3. allow user to modify that new field, where prefixes are listed

4. recollect prefixes, when template is modified (check, when it's is compiled)

5. in parser:
- wrap all template content (virtually) in m_Cache tags with "key" value equals to that new field from Category table
- pages, that are not present in Category table will be never cached this way
(0001480)
Dmitry   
2010-02-16 17:22   
Ability to Cache Url Parameters

Table: CachedUrls
UrlId
Url (without base url, for debugging purposes)
Hash (crc32 of PageUrl)
Prefixes ("|prefixA|prefixB|prefixC:IDA|prefixX|")
ParsedVars (serialized vars)
CreatedOn (date, when parsed)
ExpiresOn (-1 or second count till expires)
-----------

When serial is incremented, then we delete corresponding records from CachedUrls table (by Prefixes column) using prefix and id given to incrementCacheSerialMethod
(0001485)
alex   
2010-02-24 11:26   
Fix committed to 5.1.x branch. Commit Message:

Fixes 0000107: Implement "MemCached" functionality
(0001486)
alex   
2010-02-24 11:28   
Reminder sent to: Dmitry

Testing plan:

1. Following new category fields work and can be controlled via site configs: PageCache,OverridePageCacheKey,PageCacheKey,PageExpiration.

2. Memory Cache interface in system tools section.
3. Memcached Server can be defined via "MemcacheServers" configuration variable (configuration -> website -> advanced) or via config.php file (same variable name).
4. Mod-rewrite url parse result is cached in CachedUrls table and is properly used and reset (based on parsed variables OR page expiration defined in Category.PageExpiration field).
5. Upgrade from non-memcache installation to memcache installation, upgrade from memcache installation to memcache installation.
6. Category permission cache is reset (with progress bar OR not), when installation of separate module is made.
7. Database queries in debugger contain prefix_special information below them.
8. Database queries that will be cached are marked with "Name (cached)" mark in debugger.
9. Category corresponding to current page (special is "-virtual") is loaded using simplified sql (without calculated and custom fields and without left joins).

10. List of affected methods:
in-auction/units/helpers/ebay_helper.php - CheckItemsLimits, CheckCallLimits, ClearConfigCache
in-auction/units/sections/ebay_eh.php - GetRequestInterval, OnGetSellerList, OnGetSellerTransactions, resetAgents, SetCallCounters, SetNewToken, ClearConfigCache, OnNewTokenAccept, OnAuthorizeForToken
in-auction/units/sections/ebay_tp.php - PrintCallsArray
in-commerce/units/currencies/currency_rates.php - GetRatesData, GetRate
in-commerce/units/products/products_item.php - getPrimaryPricing
in-commerce/units/shipping_quote_engines/shipping_quote_collector.php - GetShippingQuotes
in-link/units/link_validation/link_validation_tp.php - CategoryName
core/install/install_toolkit.php - deleteCache
core/kernel/application.php - GetDefaultLanguageId, GetDefaultThemeId, GetPrimaryCurrency, getCategoryCache, getFilename, isCachingType, incrementCacheSerial, setCache, setDBCache, getCache, getDBCache, deleteCache, deleteDBCache, registerModuleConstants, Done, HREF, BuildEnv, LoadCache, LoadStructureTemplateMapping, UpdateCache, getTreeIndex
core/kernel/db/cat_event_handler.php - prepareItemStatuses, CalculateHotLimit, getAdvancedSearchCondition
core/kernel/db/cat_tag_processor.php - CategoryName, HasAdditionalImages, LastUpdated
core/kernel/db/db_connection.php - debugQuery
core/kernel/db/db_event_handler.php - StoreSelectedIDs, SaveLoggedChanges
core/kernel/db/db_tag_processor.php - PrintList2, AddCurrencySymbol
core/kernel/db/dbitem.php - Load, setTempID, LogChanges
core/kernel/db/dblist.php - Query
core/kernel/kbase.php - PrepareFieldOptions
core/kernel/nparser/nparser.php - NParser, Run, RunMainPage, generatePageCacheKey, ParseBlock, DecorateBlock, IncludeTemplate, getCache, setCache, FormCacheKey, PushPointer, CacheStart, CacheEnd
core/kernel/nparser/ntags.php - _Tag_Cache::Open, _Tag_Cache::Close
core/kernel/processors/tag_processor.php - FormCacheKey, ProcessParsedTag
core/kernel/utility/formatters/multilang_formatter.php - LangFieldName
core/kernel/utility/temp_handler.php - UpdateChangeLogForeignKeys
core/kernel/utility/unit_config_reader.php - CacheParsedData, RestoreParsedData, ResetParsedData, includeConfigFiles
core/units/admin/admin_events_handler.php - CheckPermission, OnResetModRwCache, OnResetSections, OnResetConfigsCache, OnResetMemcache, OnMemoryCacheGet, OnMemoryCacheSet
core/units/admin/admin_tag_processor.php - CheckPermCache, MemoryCacheEnabled
core/units/categories/cache_updater.php - kPermCacheUpdater::clearData, kPermCacheUpdater::SaveData, kPermCacheUpdater::UpdateCachedPath
core/units/categories/categories_event_handler.php - prepareObject, _beforeItemChange, _resetMenuCache, LoadItem
core/units/categories/categories_tag_processor.php - getCategorySymLink, LastUpdated, SpellingSuggestions, CategoryName
core/units/custom_data/custom_data_event_handler.php - scanCustomFields, createCustomFields
core/units/forms/forms_eh.php - OnSave, OnMassDelete, _deleteSectionCache
core/units/helpers/category_helper.php - getCategoryParentPath, _getStructureTree, getTemplateMapping, getStructureTreeAsOptions
core/units/helpers/count_helper.php - resetExpiredCounters, CategoryCount, GetPermissionClause
core/units/helpers/language_import_helper.php - performImport, _performUpgrade
core/units/helpers/menu_helper.php - _prepareMenu
core/units/helpers/mod_rewrite_helper.php - processRewriteURL, _getCachedUrl, _setCachedUrl, _buildMainUrl, GetItemTemplate, getItemTemplateCustomField
core/units/helpers/permissions_helper.php - CheckUserPermission
core/units/helpers/sections_helper.php - BuildTree
core/units/helpers/skin_helper.php - compile, _getStyleInfo
core/units/helpers/themes_helper.php - refreshThemes
core/units/languages/languages_event_handler.php - CheckPermission
core/units/languages/languages_item.php - Load
core/units/permissions/permissions_tag_processor.php - CategoryPath
core/units/statistics/statistics_tag_processor.php - CountPending
core/units/themes/theme_item.php - Load
core/units/themes/themes_eh.php - CheckPermission
core/units/users/users_event_handler.php - LoadItem
(0001488)
alex   
2010-02-24 12:04   
Fix committed to 1.1.x branch. Commit Message:

Fixes 0000107: Implement "MemCached" functionality
(0001489)
alex   
2010-02-24 12:06   
Also all used phrases should be translated (category editing page, service section).
(0001508)
alex   
2010-03-02 14:20   
Patch "memcache_notification_addon.patch" doesn't show "incrementing serial ..." messages in case if memory caching is disabled.
(0001520)
alex   
2010-03-07 13:32   
Patch "memcache_theme_rebuild_fix.patch" fixes problem, that "#in-link/section#" design wasn't recognized. This happened, because I've installed In-Portal without In-Link first, and then I've installed In-Link separately. Then I've triggered theme file refresh manually (because installed doesn't do that) and it didn't give desired effect, since only general "theme" serial was reset, but current theme information was cached based on current theme's ID.
(0001521)
Dmitry   
2010-03-07 14:47   
Reminder sent to: alex

Thanks Alex.

I guess you acme across this issue accidentally?
(0001522)
alex   
2010-03-08 01:22   
Sort of. I found this when I was comparing files before merging to CVS.
(0001719)
alex   
2010-03-21 16:31   
Patch "memcache_install_fix.patch" fixes case, when "CacheHandler" is set to "Memcache" in "config.php" file and we are using "Re-install" option like this:

1. install in-portal without modules
2. re-install in-portal with in-link
3. results fatal error on front-end, when trying to get ID of "LINK.VIEW" permission.
(0001741)
alex   
2010-03-24 08:35   
Patch "memcache_install_fix_v2.patch" is improved version of "memcache_install_fix.patch" which also processes the case, when "Re-install" option is used.

Major change is that cache is reset before install is made and not after that.
(0001742)
alex   
2010-03-24 08:40   
Patch "memcache_custom_field_caching_on_install.patch" fixes problem, that custom fields were still cached during "Re-install" step, when "CacheHandler" was specified in "config.php".
(0001880)
alex   
2010-04-13 12:41   
Patch "cached_permissions_not_updated_after_group_edit.patch" (commited) solves problem, that tree sections are invisible even, when their "view" permission is set.
(0001938)
Dmitry   
2010-04-25 18:23   
(edited on: 2010-04-25 18:26)
1. Question is whether Products in Category and Products in Category Added today should be cachable. Please review the screenshot (QuestionAboutCachingNumberOfProductsInCategory.jpg)

2. Also list how I can minimize Product Details (or other item Detail) page to 7-10 SQLs per page you had before.
(0001951)
alex   
2010-04-26 04:59   
1 - they could, why not.
2 - maybe I enable full page cache, try that.
(0001955)
Dmitry   
2010-04-26 06:38   
Thanks for your reply!

1. Can you please look into that?

2. Tried and still no luck - lets look at this together.
(0002301)
Dmitry   
2010-05-23 19:48   
Reminder sent to: alex

Hi Alex,

Reviewed some more and have some additional notes:


1. Would you please change the order of Caching Settings configuration variables (Admin->Configuration->Website Advanced:

- CacheHandler
- MemcacheServers

2. Add a hit label for "MemcacheServers" configuration naming it "la_hint_MemcacheServers".


NOTE: No tests needed for this - please commit right away.


Thanks.
(0002313)
alex   
2010-05-24 15:03   
Patch "memcache_addon.patch" (already commited) and fixes 0000107:0002301.
(0002314)
alex   
2010-05-24 15:05   
Fix committed to 5.1.x branch. Commit Message:

Fixes 0000107: Implement "MemCached" functionality
(0002521)
alex   
2010-07-22 15:06   
Closing, since 5.1.0 release has been released.