Alessio/Data-Coupler
2
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

[Feature/Perf] Ottimizzazioni bulk pre-discovery, batch deletion sync e supporto OLE DB / Salesforce client_credentials

Browse Source 
## Bulk Pre-Discovery e riduzione query SQLite/SOQL

### KeyAssociationService — FindAssociationsByKeyValuesBulkAsync (nuovo)
- Aggiunta query bulk 'WHERE KeyValue IN (...)' per recuperare N associazioni con 1 sola query SQLite
  (chunking a 500 chiavi per rispettare il limite ~999 parametri di SQLite)
- Aggiunta interfaccia IKeyAssociationService e delegata in DataConnectionCredentialService / IDataConnectionCredentialService

### AssociationService — BatchFindOrCreateAssociationsAsync (nuovo)
- Nuovo metodo bulk che sostituisce i loop per-record durante l'analisi composite:
  1) 1 query SQLite bulk per tutte le chiavi
  2) Per le chiavi non trovate: SOQL 'IN (...)' su Salesforce in chunk da 200 via BatchExecuteQueriesAsync
     (ceil(K/25) HTTP Composite call invece di K singole)
  3) Salvataggio parallelo delle associazioni pre-discovery scoperte
- Fallback per-record automatico per client REST non Salesforce
- Aggiornata interfaccia IAssociationService con documentazione XML completa

### DataCoupler.razor.cs — STEP A/B nel flusso COMPOSITE
- Pre-Discovery spostata FUORI dal loop parallelo (STEP A, prima dell'analisi)
- associationsByKey pre-popolato con BatchFindOrCreateAssociationsAsync
- STEP B: il loop parallelo usa TryGetValue O(1) invece di query async per record
- Rimozione blocco ~40 righe di per-record lookup / fallback duplicati

## Salesforce Composite API — Batch Delete e Patch

### SalesforceServiceClient — metodi batch (nuovi)
- BatchDeleteEntitiesAsync: elimina N record con ceil(N/25) Composite call invece di N
- BatchPatchSingleFieldAsync: aggiorna un singolo campo su N record tramite BatchUpdateEntitiesAsync

### DeletionSyncService — refactoring batch
- ExecuteBatchedSalesforceDeletionsAsync: orchestrazione batch per Delete / Deactivate / Mark su Salesforce
- ExecuteSequentialDeletionsAsync: loop sequenziale esistente estratto in metodo riutilizzabile
- Dispatcher: Salesforce -> batch Composite, altri client REST -> sequenziale

## Supporto OLE DB (database)

### DatabaseSchemaProviderFactory
- Aggiunto case DatabaseType.OleDb -> new OleDbSchemaProvider() nel factory switch

### DatabaseMethod.cs
- Aggiunto metodo IsOleDbConnection() (parallelo a IsOdbcConnection())
- Query validation e manager temporaneo estesi a OLE DB oltre che ODBC
- GetLimitedQuery: aggiunto case OleDb -> 'SELECT TOP N FROM (subquery)'

## Salesforce OAuth2 — fix client_credentials

### CredentialService.cs
- Aggiunto 'GrantType' alla HashSet serviceSpecificKeys per preservarlo nella serializzazione AdditionalParameters

### DataConnectionCredentialService.cs
- Refactored BuildRestServiceOptions in helper statico riutilizzato da entrambi i metodi GetRestServiceOptions
- Mapping coerente ClientId/ClientSecret/GrantType per Salesforce (allineato a DataConnectionFactory)
- TestSalesforceOAuthLogin: branch esplicito per client_credentials (no username/password/token)
  con validazione preventiva ClientId+ClientSecret obbligatori
- Log flow label (password|client_credentials) in tutti i messaggi di autenticazione

## VS Code tasks

### .vscode/tasks.json
- Rimosso task generico 'Publish Data_Coupler'
- Aggiunti due task separati: win-x64 e win-x86, entrambi SingleFile + Self-Contained + ReadyToRun
This commit is contained in:
Alessio Dal Santo
2026-05-28 11:15:18 +02:00
parent 82e0d6bc77
commit 344853fde9
13 changed files with 886 additions and 223 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DataConnection/CredentialManagement/Interfaces/IDataConnectionCredentialService.cs
+8
View File
@@ -85,6 +85,14 @@ public interface IDataConnectionCredentialService
Task<KeyAssociation?> FindKeyAssociationByValueParallelAsync(string keyValue);
Task<bool> DeleteKeyAssociationParallelAsync(int id);
/// <summary>
/// Bulk lookup associazioni - una sola query SQLite per N chiavi.
/// </summary>
Task<Dictionary<string, KeyAssociation>> FindKeyAssociationsByValuesBulkAsync(
IEnumerable<string> keyValues,
string destinationEntity,
string restCredentialName);
// Deletion synchronization operations
Task<int> MarkDeletedAssociationsAsync(List<string> sourceKeyValues, string destinationEntity, string restCredentialName);
Task<List<KeyAssociation>> GetPendingDeletionsAsync(string destinationEntity, string restCredentialName);
DataConnection/CredentialManagement/Services/DataConnectionCredentialService.cs
+83 -41
View File
@@ -168,16 +168,7 @@ public class DataConnectionCredentialService : IDataConnectionCredentialService
if (credential == null)
throw new InvalidOperationException($"REST API credential '{credentialName}' not found");
var options = new DataConnection.REST.Configuration.RestServiceOptions
{
BaseUrl = credential.BaseUrl,
ApiKey = credential.ApiKey,
Username = credential.Username,
Password = credential.Password,
AuthToken = credential.AuthToken,
TimeoutSeconds = credential.TimeoutSeconds,
IgnoreSslErrors = credential.IgnoreSslErrors
};
var options = BuildRestServiceOptions(credential);
_logger.LogDebug("Created RestServiceOptions for credential: {Name} ({BaseUrl})",
credentialName, credential.BaseUrl);
@@ -191,19 +182,42 @@ public class DataConnectionCredentialService : IDataConnectionCredentialService
if (credential == null)
throw new InvalidOperationException($"REST API credential with ID '{credentialId}' not found");
var options = BuildRestServiceOptions(credential);
_logger.LogDebug("Created RestServiceOptions for credential ID: {Id} ({BaseUrl})",
credentialId, credential.BaseUrl);
return options;
}
private static DataConnection.REST.Configuration.RestServiceOptions BuildRestServiceOptions(RestApiCredential credential)
{
var options = new DataConnection.REST.Configuration.RestServiceOptions
{
BaseUrl = credential.BaseUrl,
ApiKey = credential.ApiKey,
Username = credential.Username,
Password = credential.Password,
AuthToken = credential.AuthToken,
TimeoutSeconds = credential.TimeoutSeconds,
IgnoreSslErrors = credential.IgnoreSslErrors
};
_logger.LogDebug("Created RestServiceOptions for credential ID: {Id} ({BaseUrl})",
credentialId, credential.BaseUrl);
// Mapping coerente con DataConnectionFactory.CreateRestServiceClientAsync
switch (credential.ServiceType)
{
case RestServiceType.Salesforce:
options.ApiKey = credential.ClientId;
options.AuthToken = credential.ClientSecret;
options.SalesforceGrantType = credential.GrantType;
break;
case RestServiceType.SapB1ServiceLayer:
options.ApiKey = credential.CompanyDatabase;
options.AuthToken = credential.AuthToken;
break;
default:
options.ApiKey = credential.ApiKey;
options.AuthToken = credential.AuthToken;
break;
}
return options;
}
@@ -550,8 +564,8 @@ public class DataConnectionCredentialService : IDataConnectionCredentialService
{
try
{
_logger.LogInformation("Testing Salesforce authentication for {Name} ({BaseUrl})",
credential.Name, credential.BaseUrl);
_logger.LogInformation("Testing Salesforce authentication for {Name} ({BaseUrl}, GrantType={GrantType})",
credential.Name, credential.BaseUrl, credential.GrantType);
_logger.LogDebug("Salesforce credential details: Username={Username}, HasPassword={HasPassword}, HasSecurityToken={HasSecurityToken}, HasClientId={HasClientId}, HasClientSecret={HasClientSecret}",
credential.Username, !string.IsNullOrEmpty(credential.Password), !string.IsNullOrEmpty(credential.SecurityToken),
@@ -560,49 +574,69 @@ public class DataConnectionCredentialService : IDataConnectionCredentialService
using var httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromSeconds(credential.TimeoutSeconds);
// Test di autenticazione OAuth2
var tokenUrl = credential.BaseUrl.TrimEnd('/') + "/services/oauth2/token";
var tokenData = new List<KeyValuePair<string, string>>
{
new("grant_type", "password"),
new("username", credential.Username ?? "")
};
List<KeyValuePair<string, string>> tokenData;
string flowLabel;
// Aggiungiamo password + security token se disponibile
var password = credential.Password ?? "";
if (!string.IsNullOrEmpty(credential.SecurityToken))
if (credential.GrantType == CredentialManager.Models.SalesforceGrantType.ClientCredentials)
{
password += credential.SecurityToken;
}
tokenData.Add(new("password", password));
// Client Credentials flow — server-to-server, no user
if (string.IsNullOrEmpty(credential.ClientId) || string.IsNullOrEmpty(credential.ClientSecret))
{
return (false, "Flusso client_credentials richiede ClientId e ClientSecret configurati.");
}
// Aggiungiamo client credentials se disponibili
if (!string.IsNullOrEmpty(credential.ClientId))
{
tokenData.Add(new("client_id", credential.ClientId));
tokenData = new List<KeyValuePair<string, string>>
{
new("grant_type", "client_credentials"),
new("client_id", credential.ClientId),
new("client_secret", credential.ClientSecret)
};
flowLabel = "client_credentials";
}
if (!string.IsNullOrEmpty(credential.ClientSecret))
else
{
tokenData.Add(new("client_secret", credential.ClientSecret));
// Password flow (default)
var password = credential.Password ?? "";
if (!string.IsNullOrEmpty(credential.SecurityToken))
{
password += credential.SecurityToken;
}
tokenData = new List<KeyValuePair<string, string>>
{
new("grant_type", "password"),
new("username", credential.Username ?? ""),
new("password", password)
};
if (!string.IsNullOrEmpty(credential.ClientId))
{
tokenData.Add(new("client_id", credential.ClientId));
}
if (!string.IsNullOrEmpty(credential.ClientSecret))
{
tokenData.Add(new("client_secret", credential.ClientSecret));
}
flowLabel = "password";
}
_logger.LogDebug("Posting to Salesforce token URL: {TokenUrl}", tokenUrl);
_logger.LogDebug("Posting to Salesforce token URL: {TokenUrl} (flow={Flow})", tokenUrl, flowLabel);
var tokenContent = new FormUrlEncodedContent(tokenData);
var response = await httpClient.PostAsync(tokenUrl, tokenContent);
if (response.IsSuccessStatusCode)
{
var responseContent = await response.Content.ReadAsStringAsync();
_logger.LogInformation("Salesforce authentication successful for {Name}", credential.Name);
return (true, $"Autenticazione Salesforce riuscita!\n\nDettagli:\n- Login URL: {credential.BaseUrl}\n- API Version: {credential.ApiVersion}\n- Sandbox: {credential.IsSandbox}\n- Tipo Auth: OAuth2\n- Timeout: {credential.TimeoutSeconds}s");
_logger.LogInformation("Salesforce authentication ({Flow}) successful for {Name}", flowLabel, credential.Name);
return (true, $"Autenticazione Salesforce riuscita!\n\nDettagli:\n- Login URL: {credential.BaseUrl}\n- API Version: {credential.ApiVersion}\n- Sandbox: {credential.IsSandbox}\n- Tipo Auth: OAuth2 ({flowLabel})\n- Timeout: {credential.TimeoutSeconds}s");
}
else
{
var errorContent = await response.Content.ReadAsStringAsync();
_logger.LogWarning("Salesforce authentication failed for {Name}. Status: {StatusCode}, Response: {Response}",
credential.Name, response.StatusCode, errorContent);
return (false, $"Autenticazione Salesforce fallita. Status: {response.StatusCode}\nDettagli: {errorContent}");
_logger.LogWarning("Salesforce authentication ({Flow}) failed for {Name}. Status: {StatusCode}, Response: {Response}",
flowLabel, credential.Name, response.StatusCode, errorContent);
return (false, $"Autenticazione Salesforce ({flowLabel}) fallita. Status: {response.StatusCode}\nDettagli: {errorContent}");
}
}
catch (HttpRequestException ex)
@@ -1074,6 +1108,14 @@ public class DataConnectionCredentialService : IDataConnectionCredentialService
return await _keyAssociationService.FindAssociationByKeyValueParallelAsync(keyValue);
}
public async Task<Dictionary<string, KeyAssociation>> FindKeyAssociationsByValuesBulkAsync(
IEnumerable<string> keyValues,
string destinationEntity,
string restCredentialName)
{
return await _keyAssociationService.FindAssociationsByKeyValuesBulkAsync(keyValues, destinationEntity, restCredentialName);
}
public async Task<bool> DeleteKeyAssociationParallelAsync(int id)
{
return await _keyAssociationService.DeleteAssociationParallelAsync(id);
DataConnection/DB/EF/DatabaseSchemaProviderFactory.cs
+4 -1
View File
@@ -19,7 +19,10 @@ public class DatabaseSchemaProviderFactory
{
return databaseType switch
{
DatabaseType.SqlServer => new SqlServerSchemaProvider(), DatabaseType.Odbc => new OdbcSchemaProvider(), // Aggiungere qui altri provider quando implementati
DatabaseType.SqlServer => new SqlServerSchemaProvider(),
DatabaseType.Odbc => new OdbcSchemaProvider(),
DatabaseType.OleDb => new OleDbSchemaProvider(),
// Aggiungere qui altri provider quando implementati
// DatabaseType.MySql => new MySqlSchemaProvider(),
// DatabaseType.PostgreSql => new PostgreSqlSchemaProvider(),
// DatabaseType.Oracle => new OracleSchemaProvider(),
DataConnection/REST/Implementations/SalesforceServiceClient.cs
+168 -3
View File
@@ -126,13 +126,19 @@ namespace DataConnection.REST.Implementations
{
_accessToken = tokenResponse.AccessToken;
_instanceUrl = tokenResponse.InstanceUrl;
_tokenExpiry = DateTime.UtcNow.AddSeconds(3600); // Salesforce doesn't always return expires_in
// Se Salesforce restituisce expires_in (es. client_credentials), usalo con un margine di 60s;
// altrimenti (password flow non restituisce expires_in) fallback al default 1h conservativo.
var ttlSeconds = tokenResponse.ExpiresIn.HasValue && tokenResponse.ExpiresIn.Value > 60
? tokenResponse.ExpiresIn.Value - 60
: 3600;
_tokenExpiry = DateTime.UtcNow.AddSeconds(ttlSeconds);
_httpClient.DefaultRequestHeaders.Authorization =
new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", _accessToken);
_logger.LogInformation("Salesforce authentication ({Flow}) successful. InstanceUrl={InstanceUrl}, TokenExpiry={Expiry}",
flowName, _instanceUrl, _tokenExpiry.ToLocalTime());
_logger.LogInformation("Salesforce authentication ({Flow}) successful. InstanceUrl={InstanceUrl}, TokenExpiry={Expiry} (TTL {Ttl}s, server expires_in={ExpiresIn})",
flowName, _instanceUrl, _tokenExpiry.ToLocalTime(), ttlSeconds, tokenResponse.ExpiresIn?.ToString() ?? "null");
return true;
}
@@ -1616,6 +1622,13 @@ namespace DataConnection.REST.Implementations
[JsonPropertyName("signature")]
public string Signature { get; set; } = string.Empty;
/// <summary>
/// Durata di validità del token in secondi. Presente solo in alcuni flussi OAuth Salesforce
/// (es. client_credentials, JWT bearer); assente in altri (es. username/password) — in tal caso 0/null.
/// </summary>
[JsonPropertyName("expires_in")]
public int? ExpiresIn { get; set; }
}
private class SalesforceSObjectsResponse
@@ -2161,5 +2174,157 @@ namespace DataConnection.REST.Implementations
}).ToList();
}
}
/// <summary>
/// Elimina N record SObject in batch tramite Salesforce Composite API.
/// Riduce N HTTP calls a ceil(N/25), eseguite in parallelo.
/// </summary>
/// <param name="entityName">Nome SObject (es. "Account").</param>
/// <param name="entityIds">Lista di Id da eliminare.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>Risultato per ogni Id (Success/ErrorMessage).</returns>
public async Task<List<CompositeOperationResult>> BatchDeleteEntitiesAsync(
string entityName,
List<string> entityIds,
CancellationToken cancellationToken = default)
{
if (!IsAuthenticated())
{
_logger.LogDebug("Error: Not authenticated to Salesforce. Cannot perform batch delete.");
return new List<CompositeOperationResult>();
}
if (entityIds == null || entityIds.Count == 0)
return new List<CompositeOperationResult>();
const int maxBatchSize = 25;
var batches = new List<(List<string> batch, int startIndex, int batchNumber)>();
for (int i = 0; i < entityIds.Count; i += maxBatchSize)
{
var chunk = entityIds.Skip(i).Take(maxBatchSize).ToList();
batches.Add((chunk, i, (i / maxBatchSize) + 1));
}
_logger.LogDebug($"--- BatchDelete: {entityIds.Count} record in {batches.Count} batch (parallel) ---");
var batchTasks = batches.Select(async b =>
await ExecuteDeleteBatchAsync(entityName, b.batch, b.startIndex, cancellationToken));
var batchResults = await Task.WhenAll(batchTasks);
var allResults = new List<CompositeOperationResult>();
foreach (var r in batchResults) allResults.AddRange(r);
_logger.LogDebug($"All delete batches completed: {allResults.Count(r => r.Success)} success, {allResults.Count(r => !r.Success)} failed");
return allResults;
}
private async Task<List<CompositeOperationResult>> ExecuteDeleteBatchAsync(
string entityName,
List<string> batch,
int startIndex,
CancellationToken cancellationToken)
{
try
{
var compositeUri = $"{_instanceUrl}/services/data/v60.0/composite/";
var compositeRequest = new SalesforceCompositeRequest();
for (int i = 0; i < batch.Count; i++)
{
var entityId = batch[i];
compositeRequest.CompositeRequest.Add(new SalesforceCompositeSubRequest
{
Method = "DELETE",
Url = $"/services/data/v60.0/sobjects/{entityName}/{entityId}",
ReferenceId = $"delete_{startIndex + i}"
// Body intenzionalmente null per DELETE
});
}
var jsonContent = new StringContent(
JsonSerializer.Serialize(compositeRequest, SalesforceJsonOptions),
System.Text.Encoding.UTF8,
"application/json");
var response = await _httpClient.PostAsync(compositeUri, jsonContent, cancellationToken);
if (!response.IsSuccessStatusCode)
{
var errorContent = await response.Content.ReadAsStringAsync(cancellationToken);
_logger.LogDebug($"Salesforce Batch Delete failed: {response.StatusCode} - {errorContent}");
return batch.Select((id, idx) => new CompositeOperationResult
{
ReferenceId = $"delete_{startIndex + idx}",
EntityId = id,
Success = false,
ErrorMessage = $"Batch operation failed: {response.StatusCode} - {errorContent}"
}).ToList();
}
var responseContent = await response.Content.ReadAsStringAsync(cancellationToken);
var compositeResponse = JsonSerializer.Deserialize<SalesforceCompositeResponse>(responseContent, SalesforceJsonOptions);
var results = new List<CompositeOperationResult>();
if (compositeResponse?.CompositeResponse != null)
{
for (int i = 0; i < compositeResponse.CompositeResponse.Count; i++)
{
var sub = compositeResponse.CompositeResponse[i];
var originalId = i < batch.Count ? batch[i] : string.Empty;
var result = new CompositeOperationResult
{
ReferenceId = sub.ReferenceId,
EntityId = originalId,
HttpStatusCode = sub.HttpStatusCode,
// 204 No Content è la risposta di successo standard per DELETE
Success = sub.HttpStatusCode >= 200 && sub.HttpStatusCode < 300
};
if (!result.Success)
result.ErrorMessage = sub.Body?.ToString() ?? "Unknown error";
results.Add(result);
}
}
return results;
}
catch (Exception ex)
{
_logger.LogDebug($"Error during Salesforce batch delete: {ex.Message}");
return batch.Select((id, idx) => new CompositeOperationResult
{
ReferenceId = $"delete_{startIndex + idx}",
EntityId = id,
Success = false,
ErrorMessage = ex.Message
}).ToList();
}
}
/// <summary>
/// Bulk PATCH per aggiornare un singolo campo (es. campo di tombstone/mark-as-deleted) su N record.
/// </summary>
public Task<List<CompositeOperationResult>> BatchPatchSingleFieldAsync(
string entityName,
IEnumerable<string> entityIds,
string fieldName,
object fieldValue,
CancellationToken cancellationToken = default)
{
var updates = entityIds
.Where(id => !string.IsNullOrEmpty(id))
.ToDictionary(
id => id,
_ => (Dictionary<string, object>)new Dictionary<string, object> { { fieldName, fieldValue } });
return BatchUpdateEntitiesAsync(entityName, updates, cancellationToken);
}
}
}